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A CONCEPTUAL FRAMEWORK FOR EDUCATIONAL OUTCOMES 



Introduction 

Let us examine a hypothetical scenario. James Green • eceived a degree 
in business administration from Alpha College in the mid 1960s. He had spent 
five years at the college— three as a full-time student and two more as a 
part-time evening student. During that time, Jim was exposed to a variety 
of opportunities and experiences that not only increased his knowledge and 
skills, but also affected his values, relationships with peers, and other 
personal characteristics. He was sophomore class representative to the 
student senate, and later served as the president of the college's business 
students club. Jim was the first in his family to receive a college degree, 
and that achievement had an impact on his parents' social status. It also 
strengthened the determination of his younger sister and brother to attend 
college, which they eventually did. 

Shortly after graduating from Alpha, Jim joined a firm headed by a 
classmate's father. He enjoyed good pay, status, security, and excellent 
opportunities for advancement. The company benefited £rom Jim's college- 
enhanced abilities, and his ideas led to significant operating economies. 
One year after graduation, Jim married Sarbara Smith, whom he had met while 
attending Alpha College. Barbara also graduated from Alpha and after some 
graduate training, successfully pursued a career in nursing. At the same 
time, with encouragement from his company through released time, Jim was 
completing the masters degree program in management science that Alpha 
offered. They both appreciated the security that their professional status 
permitted. Later, as parents of two children, they came to the realization 
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that their college experiences were affecting the way that they related 

to their children—for example, they made greater attempts than would otherwise 

have been the case to instill openness to new ideas, a respect for others 

no matter who they might be, and a desire to seek out new situations and 

be adaptable. 

Jim participated in civic and community affairs, partially because of 
an interest stimulated by a government course he had taken as an undergraduate 
at Alpha College. He served on the local school board, the city council, 
and the board of a local bank. He was also active in several professional 
associations, made numerous state and national presentations, and eventually 
became a national officer of one group. Jim continued to support Alpha 
College financially through its alumni fund and served as an alumni contact , 
for prospective students interested in Alpha's business programs. In addition, 
because of the civic and professional contacts he made, he served as a 
consultant to the U.S. Commerce Department and several foreign governments, 
relating to the development of efficient management information systems. 

It is obviously difficult to separate the effects of his Alpha College 
experiences from other factors impinging on the life of James Green. Furthermore, 
Alpha College not only affected him directly but also, through him, apparently 
impacted many others, including his parents, his brother and sister, his 
fellow students, his wife and children, his company, his city and various 
community and civic organizations, Alpha College itself, his professional 
associates, his state, his country, and even foreign countries. In addition 
to impacts through students like Jim, Alpha College also was directly 
affecting groups, communities, and individuals who were not students, through 
research conducted by the faculty, an extension advisory service, a day-care 
program set up for working mothers, weekend gynuasium privileges for community 
youth, a concert series for the community, goods and services purchases from 
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local businesses, and similar activities. 

Educational-outcome scenarios of this kind could be developed for any 
college or university, and for elementary and secondary education as well. 
This example was generated using a conceptual framework for educational outcomes 
developed in recent years at the National Center for Higher Education Management 
Systems (NCHEMS) . Work on this framework took into account the fact that 
although most people have an intuitive idea of what an educational outcome 
is, widely different concepts of outcome appear in the literature and else- 
where. Outcomes have been equated by some to such conceptually different terms 
as efficiency, productivity, effectiveness, benefits, output level, value 
added, impacts, and performance. There did not appear to be a. generic con- 
ceptual framework that defined educational outcome in a generally accepted and 
operationally useful manner. Therefore, NCHEMS staff began in 1974 to synthesize 
the extensive literature pertaining to the issue (Lenning 1977b) and to 
develop a framework that would have general acceptance throughtout postsecondary 
education. This conceptual framework became the basis for the NCHEMS Outcomes 
Structure, a three-dimensional system for organizing outcomes information for 
purposes of classification, analysis, and decisionmaking (Lenning, Lee, Micek, 
and Service 1977; Lenning 1977a). 

The remainder of this paper explicates the conceptual framework for 
educational outcomes developed at NCHEMS . Although it was specifically 
developed with postsecondary education in mind, the framework may also be 
relevant to educational outcomes at other levels. The basic elements of any 
educational outcome are discussed first. This is followed by a presentation 
of other factors important for an in-depth understanding of particular edu- 
cational outcomes. 
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The Concept of Educ~ rial Outcomes 

The approach taken with respect to defining a general and operationally 
useful concept of educational outcomes was to identify the basic attributes 
and characteristics of any such outcomes. Six such elements were identified 
as critical to defining and differentiating among educational outcomes, and 
each has been given a descriptive 4 name as shown below: 

1. Output /Impact - the degree of directness which characterizes the 
relation between educational process and educational outcome. 

2. Form - the mode or fashion in which the outcome is observed. 

3. Measurability - the degree to which the outcome can be quantitatively 
described. 

4. Change Status - the degree of modification of the b^atus quo 
associated with the outcome. 

5. Focus - the basic entity that is affected by the outcome. 

6. Neutrality - the value-free character of educational outcomes. 
These elements are elaborated upon below. 

1. Output/Impact . A major problem is that educational outputs have 
generally not been clearly distinguished from educational impacts. 
Failure to make a clear conceptual distinction between outputs and 
impacts reduces our ability to identify, organize, and analyse the 
wide range of educational outcomes. Both concepts are very Important 
and each is a type of outcome. However, as policy analysts have found 
in other types of institutions and organizations (for example, 
Easton 1965; Robinson and Majak 1967; Cook and Scioli 1972; and 
Dye 1975), it is essential that outputs and impacts be distinguished 
from one another. 



Educational outputs are the direct end products, events, or conditions 
that result from facilitation and production processes within educational 
institutions. Examples of outputs at the college level are achievement levels, 
knowledge, degrees, program completers, publications, cultural or entertainment 
events sponsored or provided, and scientific or artistic advances. Educational 
impacts , on the other hand, are the indirect products, events, or conditions 
that result from educational outputs and earlier educational impacts. Examples 

of college impacts are greater individual incomes resulting fro™ college degrees, 

higher standards of living resulting from the increased income, and a larger 

gross national product resulting from higher standards of living. The primary 

distinction between outputs and impacts is whether or not the outcome can 

be directly linked, at least in concept, to basic institutional and programmatic 

activities. Outputs may be referred to as first-order consequences , signifying 

a direct link to institutional or program activities. Similarly, impacts 

may be considered second-order consequences , because the links to institutional 

or programmatic activities are indirect, either through an output (or more 

than one output) or through the output (s) plus a chain (or chains) of earlier 

impacts that resulted from the output(s). 

Conceptually, the distinction between outputs and impacts is quite 
straightforward. In practice, however, things become rather complex and 
difficult. First, it is often impossible to ascertain cause-and-ef feet 
relationships between educational resources, activities, and outcomes, even though 
many such relationships have been hypothesized and some have been demonstrated. 
Second, from one perspective an outcome may be viewed as an output of an 
activity, and from another perspective it could be viewed as an impact of 
the same activity— for example, the development of student leaders might be 
seen as an output of the institution and as an impact of different programs 
within the institution. Third, ai.y presumed output can be divided into component 
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that must occur before the overall output can occur. This introduces 
complexity into the determination of whether a particular outcome is an output 
or an impact. Referring to the example above relating to the development of 
student leaders, the component skills and abilities that make up leadership 
ability — ability to organize, empathize, speak fluently, motivate people, etc. — 
are outputs, and they lead to the overall ability to lead. They must be in- 
tegrated within a person, however, before the over- .1 ability to lead is 
present. 

A related problem is that some entities conceived of as outputs and impacts 
are seen as inputs by others. Thus, some would consider a curricular program 
to be an output, while others would consider it to be a producing/ facilitating 
activity that leads to outputs like student skills and knowledge. The same is 
true of the development and build-up of library and other instructional resources, 
including those being developed by faculty. For the resource developers on a 
campus, tHs development could logically be considered an output. The instructor 
using those resources could, on the other hand, view them as strictly inputs 
to the instructional process, and not as outputs. 

It should be recognized that an impact is not only less direct than 
an output, but often is less immediately realized. An output occurs curing 
or at the end of the process bringing it about, while an impact can occur during 
of at any time after the process ends. Therefore, educational institutions 
generally have much less control (if any) over impacts than they have over 
associated outputs of the institution. Although significantly positive 
correlations between amount of college education and income earned have been 
noted in the research literature, income is probably affected more by prevailing 
economic condit .ons and other postgraduate factors than by the college 
attended. Few college officials would claim that their institution has 
direct, immediate control over such an outcome. 
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In summary, outcomes are either outputs or impacts, and a particular 
outcome may be considered either when viewed from different perspectives. 
Some outcomes are definitely impacts, however, no matter who within the 
institution is viewing them; and it is important for planning purposes to 
consider them as such. Furthermore, thinking in terms of both outputs and 
impacts can help one to generate more comprehensive lists of particular kinds 
of educational outcomes, once the perspective from which one is viewing 
outcomes is clarified. To reach consensus on which outcomes should be 
considered outputs and which are impacts, it is essential that the unit 
of am lysis be made clear (the outcome-production level on which attention 
is to be fixed, e.g., course*, program, institution, system of institutions). 
2. Form , The work of Schalock and his associates (1972) makes it clear 
that both outputs and impacts can take any one of three forms: 
product is a concrete entity that endures with time, such as a program 
completer, a degree, a job, a book, or an invention. An event is 
an observable, tangible transaction or set of behaviors that does not 
endure witii time, ouch as a public seminar, a concert, or a graduation 
exercise. A condition is an intangible circumstance or set of 
circumstances, such as morale, satisfaction, an attitude or belief, 
an appreciation, social equity, or achievement. As with the output-impact 
distinction, thinking about kinds of outcomes in terms of products, 
events, and conditions can be useful for generating lists of specific 
outcomes, for developing measures or indicators of those outcomes, and 
for analyzing outcomes ir^onudion. 
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Measurability . The ease with which particular outcomes of an 
educational institution or program can be quantified or measured is 
related to the tangibility or concreteness of its form. However, 
measurability is not synonymous with tangibility or concreteness. 
For example, abstract and intangible constructs that arr often 
considered to be outcomes of a college education— analytical ability, 
reading comprehension, vocational readinesss, and various aptitudes 
can be measured in quantifiable terms. 

Determining whether specific outcomes and types of outcomes are easy 
or difficult to measure, and assessing the validity and reliability 
of their measures, can contribute to a better understanding of those 
outcomes and to any analyses that are done of them. Gross (1973) , for 
example, has broken outcome goals for five target populations (society, 
individuals, employer, government, and institutions) into these that 
are easy to measure and those that are difficult to measure. One 
problem here is that what is easy to measure in the view of one person, 
based on the availability of a particular measure, may be considered 
diff-'cult to measure by another person who considers that measure to 
be invalid. In addition, as technological advances in the measurement 
field cecur, some outcomes currently considered difficult to neasure 
may become easier to measure. 

Change Status , Another important characterise* .c educational 
outcomes is whether they are concerned with maintenance or change. 
Maintenance involves stabilization, reproduction, preservation, or 
other status quo outcomes. Examples include the continuation of 
traditions into the next generation, preservation of cultural values, 
restoration of community artifacts and paintings through guidance from 
university art students, skill maintenance provided by in-service 

12 14 



education, or maintenance of the educational level of a family. 
Conversely, change involves modification, revision, replacement 
or other alteration of the status quo. Examples include achievement 
of a college degree, greater economic and social mobility, increased 
knowledge and skill level, -ew art forms developed by college graduates, 
technological innovations, or medical discoveries. Derivation of 
these categories is based on the work of Derr (1973) and Parsons 
(1951). All educational outcomes can be thought about in these 
terms. Educational goals are designed either to preserve, replenish, 
reproduce, and stabilize the status quo or to modify, enrich, 
restructure, revise, or replace what is current. 
Focus . Still another important characteristic of an educational 
outcome concerns the specific "what" on which the maintenance or change 
is focusiag. For example, knowing that the outcome involves a 
change in knowledge and understanding, values, skills, habits, 
standards, economic conditions, or the gross national product is more 
useful than simply knowing that the outcome involves a change in 
status. Figure 1 presents the large array of focus categories and 
subcategories included as part of the "type-of-outcome"' dimension of 
the NCHEMS Outcomes Structure. These categories are based on work 
by a large number of researchers and theorists, as outlined in Lenning, 
Lee, Micek, and Service (1977, p. 27). 

Neutrality . Generically, outcome is a value-neutral concept, and thus 
educational outcomes should be thought of as being inherently neutral 
in character. Often, outcomes are equated with benefits and outcomes 
perceived to be negative in nature are ignored. But these value 
connotations are attached by the perceiver; they are not inherently 
part of or a characteristic of the outcomes. For planning purposes, 
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Figure 1 

FOCUS CATEGORIES AND SUBCATEGORIES IN THE 
TYPE-OF-OUTCOME DIMENSION OF THE MCHEMS OUTCOMES STRUCTITX 3 



Entttv Setng '.'amtamed 01 Changed 



tOOO ECONOMIC OUTCOMES 

11OO Economic Access and Independence Outcomes 
1110 Economic Access 

1120 Economic Flonaihty. Adantabiiity. and Security 
1 130 Income and Star.uaro of Living 

1?Q9 Economic Resources and Cos's 

1210 Economic Costs and Ef hc<encv 

1220 Economic Resources (inducing employees) 

1300 Economic Production 

1310 economic Productivity and Production 
13*0 Economic Services Provided 

1400 Otner Economic Outcomes 



2000 HUMAN CHARACTERISTIC OUTCOMES 

2100 Aspirations 

2*10 Desires. Aims, and Goals 
2120 DiSiiHts. LiKcs. and interests 
213C Motivation or Dnvt Ltit' 
2U0 Other Aspirationai Outcomes 

2200 Competence and Skills 

2210 Academic Smiis 

2220 Citizenship and ramiiy Membership Skills 

2230 Creatua^ Skills 

2240 Expression and Communication Skills 

2250 Intellectual Skids 

2260 Interpersonal. UaQers*ip. and Organizational Skills 

2270 Occupational and Erpic/?oihty Skills 

2260 Physical ana Motor Skills 

2290 Other Skill Outcomes 

2303 f/oraie Satisfaction, and Affective Characteristics 
2.iiQ /Mtitudos and Vaiues 

2320 Beiie.'s. Commitments, anc Philosophy of Life 
2^30 Ftenngs and Emo'i^ns 
2240 Mores. Customs, ano Standards of Conduct 
2C5C Other Aife?:>ve Outcomes 

2400 Perceptual Craract en sties 

2410 Perceptual Awareness and Sensitivity 
2420 Perception of Self 
2430 Perception of Others 
2440 Perception of Things 
2453 Otter Perceptual Outcomes 

2500 Pfsonality and Personal Copmg Characteristics 

2510 Ad*snturov.;ness and initiative 

2520 Autonomy and Indeoendence 

25j0 Dependability and Respons.cihty 

2540 Dogmatic/Cpen-Minded, Autnoriiar.an/Oomocratic 

2550 Flexibility and Adaptability 

2560 HaoitS 

2570 Psychological Functioning 

2560 Tolerance and Persistance 

2590 Other Personality and Personal Coping Outcomes 

26CC Physical and Physiological Characteristics 
2610 Physical Fitness and Waits 
2620 Physiological Health 
2S30 Oti. Physical or Physiological Outcomes 

2700 Status, RecoQnthon. and Certification 

2710 Completion or Ach.evement Award 
2 7 20 Credit Pccngrmon 
2730 Image, Hepuunon, or Suius 
2740 ticensinn and Cenihcation 

2750 Obtaining a JOD or Admit ,».on to a FeHowup Program 



Code Number entity (ie»ng Maintained or Cnanged 



2000 H/7MANCH4*ACrE/5/SnCOUrCOMES (continued) 
2760 Power and/or Autnonty 
2770 Job. School, or til* Success 

2760 Other Status. Recognition, and Congestion Outcomes 

2600 Social Activities and Ro»vs 

201 0 Adjustment to Re'utment 

2020 Alfiiidtions 

2e30 Avocanonal and Social Activities and Roms 

2840 Career and Vocation*! Activities and Rotes 

2650 Citizenship Actuihcs anc Roles 

2860 Family Admits and Roies 

2670 Fnendsnips and Reia»ionShips 

283C Other Activity and Roie Outcomes 

2900 Other Human Characteristic Outcomes 



3C00 KNOWLEDGE. TECHNOLOGY. ANO ART FORM OUTCOMES 

31C0 General Knowledge and Understanding 

31 10 Know. edge and Understanding of General Facts and 
Terminology 

3i 20 Knowledge and Understanding of Gone'ai Processes 
3130 Knowiedgoand Understanding of G»ne««*l Theory 
31 40 Other General Knowledge ana Understanding 
3200 Specialized Knowledge and Understanding 

3210 Knowledge ard Understanding of Specialized Facts 

and Terminology 
3220 Knowledge and Understanding of Specietized 

Processes 

3230 Knowledge end Understanding of Specialized The or/ 
3240 Other Scecfai»2o0 Knowledge and Undemanding 

3300 Research and Scholarship 

3310 Researcn and Scroiarshtp Knowledge and 

Understanding 
3320 Reseaich and SchoJarsmp Products 

3400 Art Forms and Works 

3*10 Arcnitecture 

3420 Oance 

3430 Debate and Oratory 

j440 Orama 

3450 Literature and Writing 

3460 Music 

34/0 Paintriq. Drawing, and Photography 

3<80 Sculpture 

3490 Other Fine Arts 

3500 Other Knowledge. Technology, and An Form Outcomes 



4000 RESOURCE ANO SERVICE PROVISION OUTCOMES 

4100 Provision of Facilities and Events 
4110 Provision of Facilities 
4120 Provision or Sponsorship of Events 

4300 Provision of Direct Services 
4210 Teaching 

4220 Advisory ano Analytic Assistance 
4230 Treatment. Care, and Referral Services 
4240 Provision of Gtr.er Services 

4300 Other Resource and Service Provision Outcomes 



5000 OTHER MAINTENANCE ANO CHANGE OUTCOMES 
5100 Aesthetic-Cultural Ac'ivities. Traditions, and Conditions 
5200 Organizational Format. Activity, and Operation 
",300 Other Maintenance and Change 



a Reprinted from Lenning, Lee, Micek, and Service (1977, p. 27). 
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in particular, the full range of outcomes and associated values 
should be considered. 

It should also be noced that individuals may differ in cheir 
perception of the value of a particular outcome. Even for an outcome 
generally viewed as positive in our society, there may be people who 
see it as negative. The stated outcome goals in our sociecy for schools 
and colleges are generally perceived by mosc people to be of positive 
value. However, Bowen (1974, pp. 14-15), has identified a number of 
general outcomes of college that many people consider to have negative 
value, for example, more liberal political, religious and social attitudes 
and values. 

Vhese six basic eienents together delineate our concepc of educational 
outcomes in an operationally useful form. Theoretically, outcome can be characterized 
as an itput or an impact ; takes a certain form ; is measurable to some greater or 
lesser extent; is concerned with c hange or maintenance ; has a particular focus or 
subject; and is inherently neutral in value. Any of the rich array of outcomes 
associated with our ficticious James Green, for example , could be categorized in 
terms of these six major characteristics. Attempting such a categorization, 
however, makes it clar that a number of factors not inherently part of educational 
outcomes nevertheless have important relationships with and effects upon those 
outcomes. These factors are reviewed in the next section. 
Factois Related to Educational Outcomes 

Identifying the most important outside factors associated with educational 
outcomes can be seen as a process of answering a series of straightforward questions: 

What activities, processes, or programs were implemented to bring about the 
outcome of concern? 
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Who receives or is affected by the outcome? 
Why was the outcome generation process initiated? 
Where did the outcome occur? 
When did the outcome occur? 
The factors corresponding to these questions are identified in this framework as: 

1 . Producer/ facilitator 

2. Audience 

3. Intention 

4. Functional area 

5. Time 

1. Producer /Facilitator . Even unintended college outcomes are typically 
stimulated by some causative or facilitative entities or factors 
within the institution. Knowledge about the entities influencing or 
causing an outcome is critical in any attempt to identify, classify, 
or analyze outcomes, since different types and levels of programs and 
organizational units are designed to produce particular kinds of 
outputs and impacts. For example, many of the outcomes intended for 
an introductory biology course may be different from tho^e intended 
for an advanced biology course, for a degree-oriented program in the 
biological sciences, for a biology department, or for the institution 
as a whole. 

Furthermore, what is viewed as an outcome from one viewpoint may be 
seen as an input from another perspective. For example, "graduates produced 
in college 11 constitute an outcome in the eyes of college officials, 
while business firms may regard these graduates as inputs. Thus 
it is necessary to link outcomes to the unit or entity that produces 
them in order to maintain a consistent perspective. Within higher 
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education, the programmatic activities of the college and its 
components have traditionally been divided into instruction, 
research, and public service. A commonly used expansion of this 
breakdown is the NCHEMS Program Classification Structure (Collier 
1978), which includes a range of support programs and is considered 
applicable to all types of postsecondary-education institutions. 

Neither the educational process within an educational institution 
nor the associated outcomes is totally separable into a set of 
component parts. As a result, it is difficult to determine which 
units within the institution contribute to the formation of a 
particular outcome. In addition, multiple programs or other organi- 
zational units within the institution often contribute to the same 
outcome, and their relative contributions cannot be easily ascertained. 
Institutional and program environments (other students, atmosphere, 
reputation, and so forth) also affect the outcomes produced. Similarly, 
a wide variety of methods, techniques, and tools can interact to 
constitute the process within the program or other unit. Each 
possible combination might be expected to result in a different 
educational outcome. Finally, the characteristics of the students and 
other inputs makes a difference in the outcomes attained. In short, 
a variety of complexities is associated with isolating the role of 
a specific educational producer/facilitator. Nevertheless, the more 
that is known about such entities, the greater the potential for 
understanding the resultant educationax outcomes. 
Audience . A second factor that affects educational outcomes is 
the identity of the persons, groups, organizations, or other entities 
that receive or are affected by the educational outcomes of concern. 
An educational institution ha^ the potential to influence a large 
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number of persons, groups, and communities — plus other entities, such 
as tne environment, un trie surface, this dimension may seem straightforward, 
but actually it represents a major difficulty in identifying and understanding 
educational outcomes. This difficulty results from the great com- 
plexity characterizing the individuals, groups, and communities 
directly served or affected by the outcomes of education. For 
example, Gross (1966) identified 26 major groups interested in the 
outcomes of any social system, such as education. Figure 2 
presents audience categories and subcategories that constitute the 
audience dimension of the NCHEMS Outcomes Structure. 
Intention . Specific outcomes may or may not be intended by the 
producing and facilitating units within the institution that give 
rise to them. In particular, many of the negatively viewed educational 
outcomes (for example, increased student drug use) are not expected 
by those planning the educational activity that causes or facilitates 
them. These unintended outcomes, or side effects, may occur either 
instead of or along with intended outcomes. Sometimes, previous 
experience or research may suggest that negative side effects will 
occur. One must then consider whether the benefits of the intended 
outcomes outweigh the expected negative side effects bv enough to warrant: 
proceeding with the activity or program. However, it should not 
be inferred from this, or from the fact that intended outcomes are 
almost always viewed as being desirable, that unintended outcomes 
are always undesirable, or negative. Some of the most important and 
valued outcomes of specific programs and activities can be unintended 
side effects, for example, a program designed specifically for 
information dissemination that stimulates the formation of an or- 
ganized student action group. 




Figure 2 

CATEGORIES AND SUBCATEGORIES OF THE 
AbUiT.lCL DIMENSION OF THE NCHEMS OUTCOMES STRUCTURE 3 

•Q m»^/OfOn0 DimM-This category r.f«s to prwsons or groups ol p«oons -no are direct cents of tne postsoconoary education un.t of 
concern a.HJ/or their .mmej.ata ass K.aws l«c n as lam.ly and relatives or peors 

1 1 Sfudenrs -individuals or g<oups ol Individuals wno Currently tr» enroll* m me program ,nst„uticn. or svS.cmof wiwcondwyerJucat.On 

12 fomrtr Srude/irs-ind.vidueis or grvups ol individuals *no loimtrly « 0r o enroled .n mi program, institution, or system ol poiiseconoary 
education 

1 3 f em»/y end fte*efi*es 0/ Srodenrs or former Sfutfenr s 

14 Peers end Assoc»afes 0/ Sfudenrs or farmer Sfudenf* 
15. fecu/fy 

1 6 S.'a/Y Ofner men Faculty 

17 Orncr inawanl/Qrovp C/renrs-An example would be an individual -no is none of ;n* aoove but is sorted Oy in advisory service offered 
by the college 

20 /nieresraasetf Co*n««oiliei-Th.a category relers 10 large groups that are Entitled as entities working toward a *e«-o*ef.ritf .ntcrnt of 
mwcn 

21 Pnuf entwi* Communmes-Communit^s *ne/e a ma»or purpose is financial remuneration ana p/oM-for eumpie. corporations, 
smeti businesses, and farmers 

22 assccafron CommumM* -Community where members oolong on the basis of affiliation ratner :n»n employment. 1 aCh as unions and 
professional sooattos. 

23 Go*cmm«tt Communis -Commun.t.es oes.gned to adm.n.ster government regulations and serv.ces sucn as ct, Mil state department 
of education, ano legislative commun.t.ts 

24 NMoo*mmfit*/**hC Service Commutes Qtfi* men me Msfrfufion Producing me Ourcome-NonproM serv.ce "A" 1 '"*" J** 
TS^Zl^^. icencrs. pn,antnrop»c founcat.ons, college. ,oth« man ths college prooucnn tne outcome., on. research 
organisations. 

25 fnili»ul*n or m.Mufionsf Un.t Prolong me Oufcome-The postseconda/y education institution and/or units within thai .nstitut.on tnat 
are perceived as tne producer/ facilitator of me outcomes) of concern. 

2C Omer/nreresf-8asetf Communis -An e»ampie would bean adhoccoai.uon task ro.ee of representative* hem two or more of tneabo<e 



areas. 



30 Cecgrapnrc-eesetf Commumf.es-Tn.s category refers to targe grouos defined on tn* basis of functions/ terMorm counaanes 

31 ioca/ Coriwnumly-A towntrvo cty ccunty, metrooontan areeor otner type of mcanty having particular c-uncar.es it «s not V 
resiricteo to me iega» of (ur.sdictona. -oundary Cu« ma (unci onat one .n wn.cn me -mpact of tne .njt.tut.on .s tor inoo. J t*» directly and 
pnys.caliv lett The boundaries «ni var/ -«"in tne »nst.lunon/program end outcomo 01 concern 

32 TAeSrare 

33 A P^g»on— An aggregation of states or pans of states 

34 77)4 rVefion 

35 An mfern^fjona/ Communrfy 

:6 Omer Geographic -Based Communmes-An eumpie wOgid be a -esearch discovery mat affects pi.maniy people living .n me coldest 
latitudes, or wnere it snows neaviiy 

40 Aoiregaiev of Peo0/e-Th,s category refer, to subpo^iations ol peo^e drsf.ngunned oy psrr.cu/ar cn*ncfens/*s -oaf nay md,ca/e common 
concert, needs or *enfs. Out *no do not necessarily nave a common merest of m.ss.on aro tnere.cre do not constitute communit.es 

41 AAiufy level Su6POPu/aiioM-SuBoopi:l«.ons defused accoru.ng to level of ability proficiency on general intellectual lurctiomng or 
specific shuls-lor namp.e. gifted, typical, disadvantaged, of lMited, semi-sxmed. unsmiied 

42 4geS«Opop<Wflf>ons 

43. cducafiona/ lev*/ SuPpopu/arions 

44 /nrorve Level SuOpopu/efions 

45 Orcupenon SuPpopo/tf'Ons 

46 Poysrce/ Crsaoj/'fy Cond;fron SuOpopu/e'ions 
AT Pace SuPpppu/efions 

46 Sex Su^popii'efions 
49 Omer Suci Aggrtgeres 

M Omer Judrences-Giempies *ouid be tne natural environment tnat is effected oy uni/crsityiconscrea researm (wn.-h m lurn *ouid be 
eipected to na.e impacts on auoiencos such as .n**'du*i} and com.n g n.i.es» ano ;op^l.it,ons cf an.mjis (sucn as me *n.mi.s atio-.too oy 
etiorts to keep depleted species from becoming e»tmct of by tne development of vewnnary mtKliCi^esi 

a Reprinted from Lenning, Lee, Micek, ind Service (1977), page 24. 
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The interaction between the producer/f acilitator of educational 
outcomes and the audience receiving or being affected by outcomes is 
also an important variable. Instituions supply educational goods and 
services because these are desired or demanded by various members of 
the community and larger society (or at least the institutions perceive 
a demand for the goods and services). In exchange, the institution 
and its programs receive financial and other necessary resources, 
including such nonpecuniary returns as status and praise. To be 
most effective o.n producing the diplomas, knowledge, skills, and 
other outcomes demanded by their clientele and funders, institutions 
need to know the costs of production and the impacts these goods 
and services will have on individuals and society. Such knowledge 
should help institutions obtain greater returns on the investments 
being made in them. 
4. Functional Area . The life of any individual, group, institution, or 
community can be viewed as involving several functional areas » 
and the outcomes of educational programs and institutions impinge 
on these areas. An understanding of particular educational outcomes 
can thus be facilitated by delineating each major functional area 
affected by educational institutions and programs. Here is one 
possible breakdown of outcomes by functional area: (1) economic 
(earnings, promotions, job opportunities, labor productivity, income 
distribution, growth of the national income), (2) educational/technologic? 
(degrees, reading habits, writing habits, educational level of 
society, advancement of scientific and technclogical knowledge, 
dissemination of new knowledge), (3) political (political attitudes, 
skill in evaluating political candidates, participation in civic 
activities, public policy development, election outcomes, international 
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relations; and (4) social/cultural/personal (religious attitudes 
appreciation of art, human relation skills, personality growth, 
crime rates, changes in traditional social values). 
5. Time . As Havighurst (1952) has suggested in his discussion of 

"developmental tasks," certain outcomes should be expected at particular 
points in one's educational career. Outcomes are difficult to bring 
about before the recipient is ready for them. Thus the time that an 
outcome occurs can be revealing. Duration or persistence of the 
outcome is also a time-related factor that can have importance fcr 
analytic purposes. Some outcomes are of short term — a college football 
game, for example. Other outcomes are lasting, such as development 
of a vaccine for influenza in a university department of medicine. It 
should be kept in mind, however, that the dividing line between short 
term and long term depends on situation and viewpoint. One person 
could consider an outcome that persists until one year after graduating 
from college as a short-term outcome, while another person might 
consider this same outcome to be long term. The basic point is that 
both time of occurrence and duration are important for collecting data 
about educational outcomes, in analyzing and interpreting such data, 
and to guide planning for outcomes. 
Conclusion 

The key question about the framework presented is the extent to which 
it appears to fulfill its intended role as a generally accepted and operationally 
useful basis for understanding and describing educational outcomes. Is the 
framework conceptually complete and nonredundant? If not, what are the 
significant omissions or overlaps? Can the framework be the basis for consensus 
about the nature of educational outcomes and factors associated with their 
understanding? Can the framework effe.tively support operational tasks such as 
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delineating outcome possibilities, planning for outcomes, developing outcome 
measures, analyzing and interpresting outcomes information, and communicating 
about the broad range of education outcomes? Preliminary evidence and 
experience (Lenning 1977a) indicate that we can give a positive response to 
these questions. However, much more questioning, testing, and development 
remains to be done. The entire arena is rich in scope and fraught with 
complexity. We hope that this paper takes a step toward untangling that 
complexity without compromising the attendant richness. 
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All social programs in our society are in response to assumed or perceived 
needs, although observers wonder sometimes whether it is not the needs of the 
provider (for financial support, for nonpecuniary benefits such as status and 
prestige, for survival, for growth, etc.) rather than the neads of the receiver 
of the services that primarily stimulate social services into happening. The 
concept of need is clearly an integral part of our culture. Most of the great 
litarary classics are built around needs and how they are or are not met. 
Satisfying important human needs is the central theme of almost all commercial 
advertising, political lobbying and advocacy, and educational jargon. 

Given that the concept of "need" is a primary driving force within education, 
including postsecondary education, the focus naturally turns to the analysis of 
which needs are most important, which are most feasible to meet, or which should 
receive priority attention in determining how available educational funds and 
other resources (such as staff, facilities, methodologies) should be expended. 
And during a period of projected enrollment decline and probable financial 
retrenchment, an objective analysis of needs becomes especially important for 
discerning which areas to maintain and which to cut back. Thus, the theme of 
the 1978 Forum of the Association for Institutional Research is "Balancing 
Needs and Resources." To insure such a balance, it is important to identify 
and assess effectively both the needs and the resources available. On the one 
end, however, there are serious difficulties related to identifying and assessing 
needs in postsecondary education - whether it be needs at the institutional, 
the state, or the national level. 

It is only recently that postsecondary education people, other than those 
in the community colleges, have expressed much interest in conducting formal, 
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objective studies to identify and assess needs. Fortunately, however, a 
significant body of knowledge about what has been called "Needs Assessment" 
has been developed over the last decade by educators at the elementary and 
secondary levels, from which postsecondary educators can borrow. Furthermore, 
noteworthy developmental work is also now taking place in postsecondary 
education. 

In spite of the developments that have taken place and the needs assess- 
ment models that have been developed, needs assessment is still a largely 
undeveloped area - for example, see the discussion by Witkin (1975). Many of 
the developmental efforts have been "piecemeal," and a conceptual framework 
that can tie all of the pieces together and guide practice has been missing. 
Therefore, from April through October of 1978, NCHEMS staff conducted a com- 
prehensive review of the needs assessment literature pertinent to the concerns 
of postsecondary education. The purpose of this effort was to sort out a 
comprehensible total picture regarding needs assessment and to develop a useful 
conceptual framework for this area. Unexpectedly, several hundred relevant 
literary sources were identified. This paper is based on the review of that 
comprehensive literature search (Lenning, Cooper, and Passmore, forthcoming) . 

The demand for systematic, objective, and concrete needs assessment 
information will undoubtedly increase as rational planning models become more 
widely used within postsecondary education institutions and agencies. The 
ability to objectively assess needs and to effectively translate them into 
insti ;utional and program responses will thus be expected to become increasingly 
more important in the years ahead. It is hoped that this preliminary formu- 
lation of a conceptual framework for needs assessment will lead to increasingly 
more-refined conceptual formulations and improved assessment models that will 
be useful for needs assessment efforts in postsecondary education. 
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The Concept of Need 



A major problem in the area of needs assessment has been the lack of a 
good definition of need. Conceptions of need that are expressed in the 
various literature are not consistent, and often they are vague and nonspecific. 
Almost all needs assessment models have used a "discrepancy" definition, but 
as illustrated by Coffing and Hutchinson (1970), such a definition is too 
limited in it? focus. Scriven (1977) cites the problem in colorful terms: 

Needs assessments have been for some tiiTie the most ludicrous 
spectacle in evaluation. The usual "models" are farcical and 
decisions based on them are built on soluable sand. One sign 
of the extent of the problem is the failure to begin with a 

tolerable definition of need Is a need a discrepancy between 

the actual and the ideal (a formula I used to like)? No, because 
we often need to improve and know how to, without knowing what 
the ideal would be like. There is some attraction about adding 
the requirement that x must be feasible, since it seems odd to 
say that one could need something that wasn't possible. But 
that would eliminate the motivation for, e.g., medical break- 
throughs [P. 25] 



Different Aspects of the Concept of Need 

Needs are viewed in different ways by those in the various disciplines. For 
example, in the fields of biology, ohysiology, and medicine needs are interpreted in 
terms of what will contribute to the efficient and effective functioning, and 
the survival and growth, of the human organism. Educators also tend to view 
needs in terms of individuals, but the focus here is more often on effective 
and efficit t functioning, survival, and growth within the community or society. 
In psychology needs are largely interpreted in terms of the perceptions of individuals. 
Psychologists usually view need as a learned construct (taught or based on 
natural experience) used to indicate a perception of disequilibrium or unsatis- 
factory condition for which pressure/need exists to right, the situation. 
Another way to state this is that need is a personal tension and means appro- 
priate for meeting a desirable goal or condition, as perceived by an individual. 
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Some psychologists would broaden this to include groups of people, and they 
construe it as a force that pressures a person or a group to reduce or 
eliminate the discrepancy between what is perceived as desired and what per- 
ceptions or experiences indicate is currently the case. Sociologists, in 
turn, focus more on groups and society. They see needs as indicators of 
problems that must be solved, plus types and levels of competence and roles 
(and their integration) that must occur, for individuals, groups, and organi- 
zations to function effectively as social units, and within a social community 
or society at large. 

Ail of the above are legitimate types of needs that must be included in 
any generic definition of needs for use in postsecondary education. The 
discrepancy definition of need guiding almost all formal needs assessment 
efforts and models up until now - the amount of discrepancy or gap that must 
be filled, through increased fulfillment or lowered threshholds of desirability, 
in order to bring the actual level of fulfillment (in terms of processes, 
procedures, conditions, outcomes, or results) up to the ideal level or con- 
dition - does not meet this condition. Neither does Coffing and Hutchinson's 
(1974) proposed alternative that need is a desired condition or state that 
may or may not be the current condition. Scriven (1977) was also bothered by 
the commonly accepted discrepancy concept of need, and proposed a formula as 
a definition: 

z needs x * z would (or does) significantly benefit from 
x and z is now (or would be, without x) in an unsatisfactory 
condi tion. (p. 25). 

To illustrate this definition, let us suppose that z represents a college 
student, and x represents the particular knowledge and skills necessary to 
obtain a job. If we say that the student NEEDS the knowledge and skills 
in order to obtain a job, we mean that: 
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(a) the student would (or does) significantly benefit from the 
knowledge and skills, 

(b) the student is now (or would be, without the krowledge and 
skills) in an unsatisfactory condition. 

Scriven's definition adds important new clarifications, as he points out in 
his rationale: 

... at least it avoids the usual fallacies of a definition- 
explicit or implicit— of need in terms of wants or preferences 
(children may need a cavity filled but they certainly don't 
want it done; conversely, people may think they need laetnle 
or CAI with Braille keys but it doesn't follow that they do.) 
Do you need a million dollars? No. Would you significantly 
benefit from it? Yes. Hence we can't omit the second clause 
in the definition, which reminds us that needs are (typically) 
necessities not luxuries, (p. 25) 

Scriven above makes the important point that wants or preferences are not 
the same thing as needs. Needs may be present that people do not recognize 
because of a lack of knowledge, because the need is being fulfilled and 
there is no discrepancy, or because it is being masked by other needs that 
demand attention. Similarly, a person may want something merely so someone 
else cannot have it, for the purpose of attracting attention, or because 
others have it. A want in such a case may be an expression of needs, but 
not the need expressed directly by the want (the expressed need is not the 
real need.) Therefore, most marketing research efforts and educational needs 
assessments are incorrect when they equate opinions, expressed desires, 
wants, or demands to needs. This is not to negate the usefulness of such 
information, which may provide good indications of needs that are present, 
and, especially if the wants are referred to by respondents in severe and 
critical terms (Taylor, Vinebery and Rufford, 1974.) But equating 
wants to needs causes people to not look for other types of info- 
mation that could confirm whether those wants are valid and reliable 
indicators of need. 
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A New Definition of Need 

All of the definitions mentioned in the preceding section are legiti- 
mate concepts of need, and each define* a particular kind of need. Thus, 
what is called for is a definition that is broad or generic enough to include 
all of those specific types of need and show how they relate to one another. 
Lenning, Cooper, and Passmore (forthcoming) have proposed a new definition 
of need that they believe has some validity in this respect: 

A NEED is a necessary or desirable condition, state or 
situation— whether it be an end result that is actuality 
(met need) or a discrepancy that should be closed between 
a current or projected actuality and a necessary or highly 
desirable end result (unmet need)— as judged by a rele- 
vant person or group using multiple objective criteria 
that have been agreed upon. 

This definition is a combination of discrepancy and level of necessity, where 
the amount of need varies directly with level of necessity and inversely with 
amount of discrepancy. Therefore, both of the following statements of need 
are valid according to this definition: "our students' needs for job infor- 
mation and employer contacts are well tatcen care of by the placement office 
on this campus," but "they have a serious need for more counseling prior to 
their interviews with prospective employers." This definition is also congruent 
with Burton and Merrill's (1977) observation that solutions in cases of unful- 
filled (unmet) needs can involve both increased fulfillment and lowered 
threshholds of desirability or satisfaction. 

This definition is pertinent to all of the different types of need 
outlined in the following section. It is also pertinent whether one is 
referring to needs: of prospective or enrolled students, of the college or 
program, of faculty or staff, of the local community or of the region, of the 
state or nation, or of society at large or other entities and groups. It also 
allows persons to speak in terms of past tense, current tense, and future 
tense when talking about needs: former needs, current needs, or projected 
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needs. 

| According to this definition, it is proper to use self report of 

m wants as an indicator of need, but the self report must have been gathered 

in an objective, unbiased manner, and there must also be other supporting 
1 evidence. Multiple sources of evidence, or multiple criteria as this is 

called in the definition, will normally lead to increased assurance of actual 
I need (increased reliability and validity) if objectivity is of paramount 

■ concern when gathering each type of evidence. 

This definition still has a potential problv^n in that it does not 
I specify when the necessity or desirability becomes significant enough to be 

m classified a need, or when the discrepancy between fulfillment and unfulf illment 

• becomes significant enough to warrant that the need is partially unmet. This 

I is in fact necessary, however, if it is to be generic in nature and apply to 

all of the types of need that have been identified by different people. On 
I the other hand, the definition does indicate that this is properly determined 

m by the judgement of a relevant person or group (who is a relevant person or 

■ group depends on the situation) using multiple, objective, agreed upon criteria 
I (who must reach agreement is not specified, but once again it varies with the 

situation) . 

Types of Outcomes for Whom 
I If one is going to attempt to identify and assess needs, it is important 

m to be very clear about whose needs are of concern. The tendency of needs 

' assessors has been to not be specific enough about ..hose needs are being 

I identified and analyzed, and to not separately consider the needs of specific 

subgroups. Similarly, needs assessors too often do not deliniate ahead of 
I time which specific types of needs are of concern to them. 
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Whose Needs Are of Concern? 

As mentioned earlier, the focus of a needs assessment study can be on 
needs within the institution (for example, courses, programs, departments, 
enrolled students, faculty, or administrators) or outside of the institution 
(for example, prospective students, groups or organizations within the local 
community or the state, or society at large). It is important to delineate 
at the earliest stages of the study exactly whose needs are of concern (where 
"whose" could even include entities like organizations and the ecological 
environment) . 

A comprehensive, two-level classification of groups and entities for 
which someone in postsecondary education might want to assess needs was 
developed as a part of the NCHEMS Outcomps Structure (Lenning et al , 1977). 
It is presented in Figure 1. The focus there was on "audiences," the persons, 
groups, or other entities that could potentially receive or be affected by 
postsecondary education outcomes. Various needs assessments have been conducted 
for many of these groups and communities. 

The listing of Figure 1 does not provide the detailed third-level categories 
needed for many outcomes studies at the institutional and program levels. 
The reason additional levels of detail are not included is that any further 
subdivisions could be based on several equally valid factors, and one user 
of the Structure would want one breakdown, while another person with a different 
philosophy, problem, and context would want a second breakdown. For example, 
students within a program could be usefully subdivided into: (1) those 
majoring in the program versus those only taking courses in the program, 
(2) age groupings, (3) commuter students versus resident students, (4-) under- 
classmen versus upperclassmen, (5) groupings according to disadvantaged 

* 

status, (6) men and women, (7) groupings according to life and career goals or 
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Figure 1 

CATEGORIES OF PERSONS, GROUPS, AND OTHER ENTITIES 
OF POSSIBLE CONCERN IN ASSESSMENTS OF NEEDS 3 

to <ndrwc*ef/Oroup C/totrs-Th* category raters to persons or groups of pinoni who are direct clients of the oostsecondery education unit of 
coram and/or thee* immediate associates, seen u fsmlly om relative* or poors. 

it. Sruo**fs— irttMduals or groups of Ihdhnduaia wno currently an enrolled In tho program, institution , or systom of postsacondary aducatlon. 

t2. former Sluoeww-^Mduais or groups of individuals who formerly war* onroiiad in tha program, institution, or systam of postsacondary 
aducatlon. 

tX ^mtly and Astfeffves or Sfudanfs or former Studorm 
U Aasrs and Aaeooerea or Stuoontt or f omw Studontt 
13. *acu«y 

t6. Starr Ornar man *acu/fy 

tf . Omar indrwouef/Oroup Ctfams- An exampia would bo an individual who is nono of tho aoova Out Is sorvad by an advisory service ot'ered 
oy the cottage. 

20. Mnsresr4ae*d Commumnea—TMs category relate to «arge groupa that ana idantlfiad as anwiae woming toward a mu-dofinod mferesr or 
mono*. 

21 bifWUm Communmea-Comrminitiee whan, a major purpoaa is financial ramunaratlon and prom-lor example, corporation*, 

small oustnaasaa. and farmars. 

22. Association Coiimn^#ffaa-Corrmwit*oa wham members bsiong on ma baa* of af filiation ratnar than amoioymant . such ss umons and 
professional aociatiaa. 

23. Qovfnmmtt Commurutm-ComtrnmHto* daaignad to adminlatar govommant regulations and servtcea. auch aa city hall, itata daoanmani 
of aducaiion. and legialativecomcnun it tee. 

24. No*gomn m *it*/Pvo*c Sarwee Commnnitioa Othor man ma institution Product** ma Oufsome— Monprofit aanrica orgamuttona. auch 
aa schools, ho aetata, weiiare egonctee. philanthropic foundatlona. colicgeo (otner than tha coilaga producing tha outcomoi. and rasaarcn 
organ nations. 

25. inttttutlon or tntttMioomi urut Producing tho Outcome— Tha postsacondary aducatlon i n at it ut ton and/or units within that institution that 
ara parcaivad aa tha producar/ facilitator of tha outcomai a) of concam 

2S. Omar frrfarssf flaaarf Commumfiee-An example would be an ad noo coalition task torca of rapraaantattvaa from two or mora of tha aoova 
araaa. 

X. Gecwme-aaaa* Commim/wee-Thta catagory refers to larga groups daf inad on tha beats of function* totritori*t ooundorm. 

31 local Community— A townemp. city, county, metropolitan ansa . or othar typa ol locality having particular boundariaa. It is not nacaasariiy 
restricted to thaiagai or luriadtcttonai boundary, out tha functional or* in which tha impact of tha institution 1a tor should oo> directly and 
pnyaicaily fait. Tha boundariaa will vary with tha institution/program and outcome of concam 

32. Tho Slot* 

33. A fegroff— An aggravation of ataias or porta of statas. 

34 Tho Nation 

35. An /nramartoner Community 

36. Othot Qaognajwaj teeed Comnwmum—An example would bo a raaaarch discovery that affects primarily people living in 'he coideat 
latitudes, or where It snows heavily. 

40 Agareseree or raoa/s This category refani to auoeopuiationa of people d/s rmguisne* Oy perf/eu/er enarecrarf srrcs mar may meVcsre common 
concerns, needs, or wants, but who do not necessarily have e common internet or mlaston. ar.d thereforedo not conatitute communitiee 



41. 


4»*ry lever $<<oeopuieffona— SuOpopuisiiona defined according to (aval of a&lity/prohciani 
specific sums— for example, girted, typical, disadvantaged, or sxmad. saml-akii:ed. unaMifad 


Vf on ganeral intellectual functioning or 
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$ tfucafionar Lever Suoooouiottont 
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Income lav*/ SuOpopu/atjona 




45. 


Occupation Swopopurenons 




45. 


Phyoreoi Ditoothty Condition SuOoooutivont 
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Sex SidOooDufUOfit 
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QthorSuch Aggregerea 





90. Otfm 4 udMncea - Examples would be the natural environment that is affected by univeraity-aponsored raaaarch (which in turn would be 
expacted to have impacta on auoiencea auch as indiwduafa and communities! and peculation* of animaia (auch aa the emmau alfactad by 
at forts to seep depleted aoeetea from becoming eitinct or by thedevefc^Ticnt of veterinary medicines) 



Reprinted from Lenning, Lee, Micek, and Service (1977, p. 24), where the focus 
was on audiences - individuals, groups, communities, organizations, etc. receiving 
or being affected by particular outcomes ofgg>nern. 



aspirations levels, and so forth. Rather than provide alternative formal 
breakdowns of each second-level category, the decision was made to provide 
procedures for users of the Structure to develop their own more-detailed 
breakdowns tailored to their specific problem, concern, or other need context 
(Lenning, 1977a). Such more-detailed categories of groups are needed for 
planning needs assessment studies also. In planning a curricular program for 
students, for example, it is important to consider the special needs of 
important student subgroups. 

An additional word should be said about the needs of groups, organizations, 
and communities as compared to the neeas of individuals. Many people have assumed 
that the needs of a group are merely aggregations of the needs of individuals 
within that group. It is true that some community and class needs are aggregations 
of individual needs. For example, an aggregate need exists when more than half 
of the freshmen entering a college have reading problems. What might have been an 
individual problem has become a community oroblem calling for administrative 
action. There are group needs that are not aggregate needs, however. These 
tend to be organizational in nature, relating to the effective functioning of 
the group as a body and to survival and growth of the group. It is even 
probable that some of these group needs will conflict with certain aggre- 
gations of individual needs, in particular cases. 

What Types of Needs Are of Concern 

Before beginning a needs assessment study, and after determining whose 
needs are of concern, it is important to specify the types of needs with 
which the study will concern itself. So that the boundaries of focus are 
clearly delineated, this should be done along several type dimensions as 
discussed later in this section. 
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Most needs assessments at the elementary and secondary education level 
have focused on the needs for particular educational outcomes. Needs for 
outcomes are important in postsecondary education also, and there are many 
types of potentially important outcomes on which one could focus. Lenning 
(1977b) has reviewed the literature for categorizations of outcomes and related 
concepts such as goals, and found almost 90 of them, some focusing on outcomes 
for individuals, some on outcomes for society, and some for both. Based on 
that review and other work, a comprehensive taxonomy of types of postsecondary 
education outcomes was developed (Lenning et al, 1977, pp. 55-66), which can 
be used in planning and developing items for a needs assessment survey 
questionnaire. 

Needs for particular outcomes imply needs for process activities. For 
example, student outcomes needs may suggest a need for special methodologies, 
environments, faculty-student ratios, teaching strategies, instructors, 
innovative techniques, etc. Such process needs can also usefully be focused 
on directly, not merely inferred from assessed needs for particular outcomes. 
In addition, there are needs in postsecondary education that are less 
directly related to outcomes, and which elementary and secondary educators 
tend not to be concerned, such as needs for financial aid, needs for information 
about institutions and programs, and needs for lodging facilities. 

When assessing needs, the focus can be broad or general and diffused 
("wide-band study") or it can be concentrated or specific and detailed 
("narrow-band study"). The wide-band study will be concerned with broad 
categories of needs while the narrow-band study will be concerned with specialized 
and detailed need categorizations. Lenning, Cooper, and Passmore (forthcoming) 
have identified a number of different need type classifications, some of 
them broad-band in focus (e.g., Maslow's [1968] need hierarchy, Parsons' 
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[1950] community need categories, and Bradshaw's Taxonomy of Social Need 
[1972]) while others have a narrow-band focus (e.g., Murray's [1978] categories 
of manifest and latent needs, Kinnick's [1975] Taxonomy of Information Needs 
of Prospective Students, and the Mooney Probelem Checklist scales [Pagels, 
1973]). 

Many of the categorizations of needs that have been developed place 
needs into categories along a continuum on a particular dimension. Examples 
of such dimensions are: developmental tasks corresponding to chronological age, 
basic versus learned (or derived) needs, personal versus social problems 
resulting in needs, maintenance versus incremental needs, conscious versus 
unconscious needs, general versus specific needs, current versus projected 
needs, critical versus routine needs, instructional versus noninstructional 
needs, economic versus noneconomic needs, needs for goods or products versus 
needs for services, easy-to-measure needs versus difficult-to-measure needs, 
and short-term or short-duration needs versus long-term or long-duration needs. 
Thinking in terms of such dimensions can be helpful for determining and setting 
the appropriate and desired boundaries of focus in planning for an assessment 
of needs. Thinking in such terms can also help one in setting needs assessment 
focus priorities within those boundaries. 

Assessing Needs 

As has been discussed, one must specifically determine whose needs, and 
what types of needs for each group, are to be assessed before plans are begun 
for conducting a needs assessment study. Now some important conceptual consid- 
erations relating to the conduct of the assessment itself will be discussed, briefly. 

Models for Assessing Needs 

A number of needs assessment models have been developed for use in the 
educational setting. Most, but not all, have been discrepancy based. Those 
developed at the elementary and secondary education levels have tended to be 

37 39 



general in their focus. Thus, some of the concepts and procedures they discuss 
may be useful also at the postsecondary level, for example: Coffing and Hutchin- 
son (1974), English and Kaufman (1975), Hoepfner et al (1972), Klein et al (1971), 
Lewis (1973), New Jersey State Department of Education (1974), Read (1974), and 
the various other models reviewed by Adams (1976), Kaufman (1971), and Witkin 
(1975, 1976). Conversely, postsecondary education models have tended to be more 
diverse and specific in their focus: vocational, occupational, and continuing 
education needs (Adams, 1976; Brown, 1974; Keim and others, 1975; Put.am, 1970; 
Smith, 1968; Tucker, 1973); environmental needs (Aulepp and Delworth, 1976); 
course-level needs (Burton and Merril, 1977); community service needs (Central 
Florida Community College Consortium, 1973; Gollattscheck et al , 1976; League of 
California Cities, 1975; Selgas, 1977); needs of the handicapped student (Coffing, 
Hodson, and Hutchinson, 1974); community information and service needs (Gotsick, 
1974); overall curricular needs (Gray, 1974; Hamilton, 1973; Pagels, 1973); admin- 
istrative functioning needs (Higher Education Management Institute, 1977); pro- 
spective students' needs for institutional and program information (Kinnick and 
Lenning, 1976; Lenning and Cooper, 1978); state-level needs for career education 
(McCaslin and Lave, 1976); needs related to performance problems (Mager and Pipe, 
1970); institutional goal needs (Peterson, 1976); curricular needs in programs for 
emergency ambulance personnel (Shook, 1969); and student financial aid needs 
(the models developed by ACT and CSS). Diverse and specialized models such as 
many of those above demonstrate the importance of tailoring concepts and procedures 
to the uniqueness of the conditions and situation. For example, an assessment 
of the curricular needs in a program for emergency ambulance personnel has to be 
quite different than one to assess curricular needs in a fine arts program, even 
though they are both focusing on curricualar needs and are both using a critical 
incident technique. 

Several writers have attempted to classify needs assessment models into types 
that show relationships among them and distinguish among their emphases and 
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and characteristics. One way to classify needs assessment approaches is according 

to the purposes for which they are designed. For example, can classify approaches 

according to those focusing on planning versus those focusing on policy formulation, 

those focusing on curriculum development versus institutional goal setting, and 

those aimed at understanding problems versus those aimed merely at identifying 

problems. Lenning, Cooper, and Passmore (forthcoming) identified 30 different 

general purposes served by needs assessment as diccussed in the literature. (In 

differentiating model types, the League of California Cities [1975] grouped 

according to three broad, overall purposes: social policy, exploratory, and 

program needs assessments.) Furthermore, these purposes can vary according to 

the types of needs being assessed, whose needs are being assessed, who is doing the 

assessing and for whom, etc. Another way to classify needs assessment approacnes 
t 

is according to time of need being assessed, such as focusing on current needs 
versus focusing on projected needs (or both) or short-duration needs versus long- 
duration needs. Some additional model classification dimensions that could be useful 
are according to: population types beir.j assessed, such as Baumheier and Heller's 
(1974) five population/purpose types - secondary data analysis, general population 
surveys, service population surveys, service provider surveys, and political and 
community surveys; breadth and detail of focus, such as the "narrow band" and "wide 
band" types of studies mentioned earlier; concreteness of the data collected, such 
ac Anderson and Associates (1976) reference to "objective" and "subjective" neeos 
assessments; approaches used for collecting data and conducting analyses, such as 
the four strategy types ouJined in the New Jersey Department of Euucation (1974) 
planning handbook and Scriven's (1977) categorization of common study types; and 
how the data are interpreted, such as Kaufman's (1972) inductive, deductive, and 
classical model differentiations. 

Planning and Operational Considerations 
Collecting v*lid and reliable evidence of need(s) is a necessary and crucial 
part of every needs assessment study. For any met and unmet need, a number of 
nlevant 'ndicators and measures usually apply. Generally, some will be better 




indicators of the presence of the need than will others, and such factors as 
whose needs are being assessed can affect the validity of the indicator or 
measure. Therefore, multiple indicators and measures should be used whenever 
feasible. This gives increased assurance of validity (that ft is a real need)., 
if they all indicate the same thing about need, plus it facilitates tailoring 
the data collection system to different groups, When one measure is less valid, 
another measure may be more valid, and vice versa. 

Currently, most need surveys are administered solely to t u °. client groups 
whose needs are being assessed. It is important not to ignore client self- 
reports about their perceived needs, but other data are needed as well. Implica- 
tions about need can also be derived from client reports about such things as 
school environment, ti.v.ir peers, disappointments or dissatisfactions, successes 
and achievements, activities, problems, and complaints. Baird (1976), for example, 
discusses the importance of identifying and remedying "brass tacks." Surveys 
should be administered to relevant others for their observations and judgments, 
also. They perhaps can be more objective, and may have more experience and 
expertise in making such judgments. Profiles showing how different groups view 
the situation can be quite reveal i~g, and the pattern of similarities and 
discrepancies may significantly facilitate understanding about the needs - 

„nen outcome needs are of concern, performance measures and history (trends) 
become very important, but self- and other-report data are still desirable also. 
Other useful supplemental data include frequency counts from institutional records 
concerning such things as attendance, complaints, amount of use (and ratings) of 
services, requests received for assistance of various kinds, etc. Similarly, 
statistics from governmental and other community agencies can provide useful 
supplemental evidence for studies o f community needs. What others have found 
in similar types of institutions, programs, or locales can also be useful 
supplemental fevidence if care is taken to examine closely how the other situations 
were similar and exactly how they were different from the one of concern. 
Concerning all of these types of data that have all ready been collected (which some 
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have termed "secondary data"), although they save costs and time in addition to 
providing useful supplemental evidence, they can lead to problems if great care 
is not taken in their use, Boyd and Westfall (1972) provide criteria for determining 
when particular secondary data are okay for a particular situation and use, and 
they also discuss precautions that can help one avoid the potential pitfalls. 

Concerning data collection methods, needs assessors aenerallv limit themselves 
to several types of traditional instruments: questionnaires, paper and pencil 
tests and interviews. However, other methods that may be just as reliable and 
valid for a particular case should be considered as alternatives and supplements 
for the traditional instruments. Lenning (1978) found fifty different methods in 
the literature that were recommended for assessments of various kinds, including 
needs assessments. Yet most needs assessors never even consider such nontradi tional 
methods that have been shown to be practical, valid, reliable, and cost efficient 
for particular purposes and contexts. As with indicators and measures, and for 
the same reasons, the use of multiple data collection methods is desirable - and 
the large variety of data collection methods available can facilitate this. 

Interpretation and use of needs data are also crucial elements in a needs 
assessment study, and too often the application of needs assessment results is 
ineffective. If needs data are going to have practical impact, the users of the 
needs assessment results must be precisely identified early in the assessment 
planning process, prior to conducting the study. Input should be solicited from 
them concerning their specific concerns and what needs information will be helpful 
to them in their decision making. Once analyses are completed, brief, concise reports 
tailored to each person's information needs should be sent to them. Additional 
ways to increase the impact of the results are also available (Lenning, Coooer, 
and Passmore, forthcoming). 

Some needs assessment approaches only try to identify needs. Yet, more 
is needed: (2) a ranking of needs according to how critical they are, and (3) 
information that can help one to understand why the need occurred. In addition to 
not attempting to perform the last function above, almost all of the available 
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assessment models, even though they may rank the needs, make use of over-simple 
and ineffective decision rules that do not consider enough factors or consider 
each factor in isolation from the others. Another problem with many of the 
models is that they key so much on current goals and objectives that the results 
of such assessments are not useful for evaluating current institutional and 
program goals, for modifying or reformulating them, or for c.veloping new goals 
to meet changing conditions. Lending, Cooper, and Passmore (forthcoming) explore 
these problems 1n detail and discuss some possible solution:.. They also provide 
1n-depth and extended discussion about all of the other topics covered in this 
paper. Needs assessment clearly is a viable tool to assist administrators and 
faculty members who are concerned about meeting client ?.nd community needs. 
Howev-r, much more development in this area needs to take place before it can 
began to reach its full potential. 
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All of the major activities conducted by administrators, faculty, and 
other professional staff members at postsecondary institutions are, presumably, 
aimed in some way toward bringing about "educational outcomes." In this "age 
of accountability," college administrators and others have become especially 
concerned about concretely identifying and understanding the impacts of their 
institution on students, the community, and society. After two years of concen- 
trated effort, the National Center for Higher Education Management Systems 
(NCHEMS) developed an "outcomes structure," a new system for organizing outcomes 
and outcomes information for purposes of classification, analysis and decision- 
making (Lenning, Lee, M1cek, and Service, 1977). Preliminary tryout and review 
of this structure suggests that it has potential use 1n: (1) stimulating people 
to realize the Importance of having Information about educational outcomes; 

(2) stimulating people to think more systematically and concretely about what 
they are trying to accomplish 1n their institutions and programs (and for whom); 

(3) helping Institutional officials to Identify educational needs, develop goals, 
translate goals Into more concrete objectives, plan for the outcomes, evaluate 
the institution and Its programs*, and (4) Improving communication about outcomes 
with clientele and concerned publics; and so forth (for example, several students 
and student personnel administrators Interviewed at a couple of small colleges 
felt that the Structure could also be used to assist students 1n planning what 
they want to accomplish for themselves during college)* 

This 1s a report of one of the projects used to try out the NCHEMS Outcomes 
Structure in a preliminary way. It was conducted in the winter of 1977 by the 
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■ University of Colorado at Boulder. Over a period of several years, through 
surveys and interviews of important clientele groups, several extensive lists 

I of intended outcomes were developed by staff of the Office of the Vice Chan- 

m eel lor for Academic Affairs on the Boulder Campus that were comprehensive for 

their planning needs. The NCHEMS Outcomes Structure was applied to these 

I Hsts In a way that would reveal the adequacy of the coverage of those lists- 

using step-by-step procedures that had been developed at NCHEMS (Lennlng, 1977). 

m This process revealed several outcomes areas considered to be important that 

■ had been overlooked in developing the lists. The lists were modified, along 
with the Freshman Questionnaire which had been based in part on the lists. 

I The remainder of this paper will go into detail about this project and Its 

_ results. 

| THE NCHEMS OUTCOMES STRUCTURE 

The purposes of the Outcomes Structure have been outlined in the introduc- 
I tion to the paper. The Structure consists of three dimensions along which 

_ outer is or information about outcomes can be placed and related to one another. 

■ The structure is based on a conceptual framework that defines six attributes 

I of educational outcomes in postsecondary education plus five other factors that 

are important for understanding particular outcomes (Lennlng, M1cek, and Service, 
| 1978; Lennlng, Lee, M1cek, and Service, 1979). The three dimensions of the 

■ structure are described below: 

o Audience — The "audience" dimension focuses on who or what receives or is 
I affected by the outcome of concern, or is intended to receive or be 

affected by 1t. It has five broad categories, and subcategories for 
I them, as outlined in Figure 1. 
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Type of Outcome — The "type-of -outcome" dimension focuses on whether the • 
outcome results in maintenance (stabilization, reproduction, or 
preservation) or change (reorganization, modification, revision, or 
replacement), and on t*e basic entity within the audience that is 
maintained or changed. This dimension also has five broad cate- 
gories, and each is subdivided into categories and subcategories of 
Increasingly more detail and specificity, as outlined in Figure 2. 
Standard definition- are provided for every category and subcategory 
of this dimension, along with Illustrative examples of outcomes 
measures and Indicators for each. 

Time — The "time" dimension focuses on when the outcome is expected to 

or does occur, and on how long tne outcome persists. The categories 
and subcategories for the "time" dimension that are deemed most 
appropriate vary, depending on the audience of concern, on the philo- 
sophy of the person using the Structure, and on the context in which 
the Structure 1s being used. (For example, its use at the Institu- 
tion-wide level may very well require different time categories than 
Its use at the Institutional program level.) To illustrate this, 
two quite different student outcome sets of categories that could be 
used for tha "time" dimension are shown in Figures 3 and 4. The one 
presented 1n Figure 4, 1s the time classification used by the Univer- 
sity of Colorado at Boulder to follow the chronological path taken 
by new freshmen through their educational careers. Identification 
of the times when different data should be collected is valuable in 
planning. 
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Figure 1 

CATEGORIES AND SUBCATEGORIES OF THE 
AUDIENCE DIMENSION OF THE NCHEMS OUTCOMES STRUCTURE 3 

10 **v»<fe«J/G'ou0 C/**r* -Thii CAiftgorv lOfnrs to pnrsons or group* of ptfMni who f Ol'oci ci^H of trtt postsocondary education unit of 
Cdflcmn aihJ/ut m** immauiata ssvkiuio*. such as lamiiy and raiativoa Of poors. 

11. Srudams-tndtvtduai* or groups of individuaia who currani ly art •nrotiod In iht program, institution. o# ayatam or post stconoary ortucation. 

12 foffnarSrutfanfs— 4noividu«is or groups ol individuals who lormarty wars onrotlod in tno program, insiiiution. or systam of postsocondary 
aducation. 

1 3. family and Atvirryt* of SHrifil* or form* StvtloMO 
1 4 Ptur? **d Auocfft of Sft/dOnr* or Form* Sruoonrs 

15. Fscu/rr 

16. S/a// Ornar man f acu/ry 

1 7 0m* //TtfrwoW/O/Oi/o Oiarws-An a*amp*o would bo an individual who »a nono of (ho aonvo out Is sorvtd by an advisory sorvica off arod 
by tntcootgt. 

30. m/arasr-OaMtf Communifiat -This catagory rotors to largo groupa tfiai aro WantlfloC at anlilloo working toward a mt^inmi tntorott or 
mtSSron. 

11. *>wja> tfltfpftf Commvfwfit *-Ccmmun*iioc wnoro a major purpoao la financial rornunaraiion and proftt-for aaampio, corporatlona, 
small buairotsaa, and rarmora. 

22. if soaarioA Commvnif^-Commumtias wharo momoara oolong on (ho oasis oi afflitaiion raihar than amptoymont. such at unions and 
ports* ton aJ aoaatios. 

23. Govtwtmanr Commomrios -Communities daaignad to administar gpvtrnment roguiations and sonnet*, sucn aa city hail, suit dapartrnoni 
of tducaiwn. ana lagtsiatrvo communities. 

24 /vonoov«vnmtnrar/<VO*c Strwc* Common** Otfm wan rftt tmtitutioo *oovemg m Ourcomo— Nonprofit sannca organlialion*. such 
a* school*, hospitals, weifaro aganctcs. philanthropic foundations, conogoa totner inan tna coiiago producing mo outcome), and rtaoareh 
organaationo. 

25 msimwon or in*nrunonsr WW froovemo ma Ourcomt— fnt pott secondary aducalion m«wu lion and/or units wiinm mat Instliutlon (hat 
art porcttvtd ss I no prooucar/'acjiitaior or tna outcome**) of concom. 

!t. Omar fctrtfttr- fltttd Commwwrnw -An esamgie would be an ad hoc coalition laati foreo of re p ro oantatrveo from two or mora of ine above 
araaa. 

X. G#og/apmcea**s> Communrrraa — Thia catagory rtfars (O larga groups deftnad on iht oasis of function* remroneJ ooundanet. 

31 toe*/ Community—* townsmo. city, county, metropolitan arts or otnar typo of locality having particular bounder**, ins not nactssariiy 
rtsincttd to tha legal or tut isdtcnonat boundary, out tna funciional on* in which tna imoaci or tno msiituiion ia (or should bt| cwactiy and 
physically fan Tha boundaries w*i vary «itn tha mstiiuiion/program and ouieomo of concarn. 

32. Tho Stoio 

33 A fttoron— A « aggragation of siatec or parts oi slat aa. 

34 Tho Notion 

35 An /nrtmat'oner Community 

35 Orncv Geographic- 5a**tf Community* asempie would oo a rtaoareh dlacovory that ar'ect* primarily ptopit ihrmg m tna coidaat 
loiiiudo*. or *hcre it snows heavily 

4C Ag>jtgotot oi too**— This carogory 'ttets to suopoouiatton* of poooto tfurmgwshed Oy osrftcuior chefacjensne* mar may /novear* common 
cimctrn*. needs or *anr*. but who do not iwcessaniy novo a common interest or mission, and therefore oo not ccnstiiuta commumitce. 

41 aomry lev* $wa*MOv<arfons~ SuOoopuiationa defined accoruing ro lowjt oi aomiy/proticiancy on general intaiiactual funciionlng or 
apooric *m»i«— tor «*emoto. girtad. typical. 4isad«aMtOed. or slutted, aemt-siuiitd. wnsaiiitd. 

42. Ago SvOOOOvistHH* 

43 . fdtrcaf'onel Cave* Suppopu/afrorts 
a mcom* tew* SuOPOpv/anons 

45. Occupefwi SuPpopWarron* 

46. nnyajcaf 0***omty Condrrron S uOpopo/anont 
47 *ac* Svboooviottont 

a. St* SuOpopu/arVon* 
49. Omar Sucn Aqoroqtm 

50. Ornar iuditnct*— rlsampii** would bo tho natural anvironmoni ih« is arfaciod by univor*uy.*oonsorad rostarch <which in turn *ouid bo 
•iptfciuo to rvivt mHMCf* on auO>w«co« men a* moiwdu^is ami \ > > H.uniiioai and popuio "* of amm«i* t*ucn aa iha animal* aftactod by 
flirort* to iiatpuauHttoil *p*uoa Ifnm oucummg iiwict m by thtumvi .fimam ol wjturmary mu nasi. 

a Reprinted from Lennlng, Lee* M1cek, and Service (19/7) » page 24. 
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Figure 2 

FOCUS CATEGORIES AND SU n CATEGORIES IN THE 
TYPE-OF-OUTCOME OIMENSION OF THE NCHEMS OUTCOMES STRUCTURE' 



Category 



Entity Being Maintained or Changed 



1000 ECONOMIC OUTCOMES 

1100 Economic Access and inoependence Outcomos 
1110 Economic Access 

1 1 70 economic f iu» lOiiity . Adaptability, and Security 
1 130 Income *ftd Stanosro ot Living 

1200 Economic Resources and Costs 

1210 Economic Costs and Efficiency 

1220 Economic Resources (including employees} 

1300 Economic Production 

1310 Economic Productivity and Production 
1320 Economic Services Provided 

1 400 Other Economic Outcomes 



Category 
Code Number 



Entity Being Maintained or Changed 



2000 hdMAN CHARACTERISTIC OUTCOMES 



2100 



2200 



Aspirations 

21 10 Desires, Aims, and Goals 
21 20 OisiiHcs, Likes, and interests 
2130 Motivation or Drive it vet 
2140 Oincr Aspiraiionai Outcomes 

Competence and Skins 
2210 Academic Skills 

Citirensmp and Family Member ship Skills 
OeativKy Skills 

Eipression and Communication Skills 
Intellectual Skiils 

Interpersonal. Leadership, and Organizational Skills 
Occupational and EmptoyaOility Skins 
Physical and Motor Skills 
Othw Skill Outcomes 



2300 



2400 



2500 



2220 
2230 
2240 
2250 
2260 
2270 
2280 
2?90 

Votale. Satisfaction, and Affective Characteristics 
2H0 Attitudes and Values 

Boiiefs, Commitments, and Philosophy of Life 
Feeimgs and Emotions 
Morss. Customs, and Standards of Conduct 
Other Aifoctue Outcomes 



2320 
?'»30 
2340 
2350 

Perceptual Characteristics 
2410 Perceptual Awareness ano Sensitivity 
°ercbpiion of Self 
p* -on of Others 
Perec , on of Things 
Other Perceptual Outcomes 



242C 
3430 
2440 
2450 

Personality and Personal Coping Characteristics 
2SlO Adventuroi.snsss and initiative 
Autonomy aid independence 
Dependability ano Responsibility 
Oogmuitc/Open-MtnUeo, Autnonta/.an/ Democratic 
Flexibility and Aoaptaomty 
Habits 

Psychological Functioning 
Tolerance and Persistence 
Other Personality and Personal Coping Outcomes 



2600 



2520 

:sjo 

2540 
2550 
2560 
2570 
2560 
2590 

Physical and Physiological Characteristics 

25 to Physical Fitness and l raits 

2620 Physiological Health 

2630 Other Physical or Physiological Outcomes 



2700 Status. Recognition , and Certification 
2710 Completion or Achievement Award 
2720 Credit Recognition. 
2730 Image. HvpuUtion. or Status 
2740 Licensing and Certification 

2750 Obtaining a Job or Admission to a Follow-up Program 



2000 HUMAN CHARACTERISTIC OUTCOMES (continued) 
2760 Power and/or Authority 
2770 Job, School, or Life Success 

2780 Other Status, Recognition, and Cettitic/.iion Outcomes 

2800 Social Activities and Roios 

2810 Adjustment to Retirement 

2820 Affiliations 

2830 Avocational and Social Activities and Roles 

2840 Career and vocational Activities and Roles 

2650 Citizenship Activities and Roles 

2860 Family Activities and Roies 

2670 Friendships and Relationships 

2880 Other Activity and Role Outcomes 

2900 Other Human Characteristic Outcomes 



3000 KNOWLEOGE. TECHNOLOGY. ANO ART FORM OUTCOMES 

3100 General Knowledge and Understanoing 

3110 Knowledge and Understanding of General Facts ent 

Terminology 

3120 Knowledge and Understanding of General Processes 
3130 Knowledge and Understanding of General T teory 
31 40 Other General Knowledge and Understanoing 
3200 Specialized Knowledge and Understanding 

3210 Knowledge eno Understanding of Specialized Facts 

and Terminology 
3220 Knowledge and Understanding of Specteiized 

Processes 

3230 Knowledge end Unoers tending of Specialized Theory 
3240 Other Specialized Knowledge ano Understanding 

3300 Research and Scholarship 

3310 Rsseerch and Scholarship Knowledge and 

Understanoing 
3320 Research and Scholarship Products 

3400 Art Forms end works 

3410 Architecture 

3420 Oance 

3430 Oeoate and Oratory 

3440 Or erne 

3450 Literature end Writing 

3460 Music 

3470 Painting, Orawing, and Photography 

3480 Sculpture 

3490 Other Fine Arts 

3500 Other Knowledge. Technology, and Art Form Outcomes 



4000 RESOURCE ANO SERVICE PROVISION OUTCOMES 

4100 Provision ot Faculties and Events 
4110 Provision ot Facilities 
4120 Provision or Sponsorship of Events 

4200 Provision of Oirect Services 
4210 Teacnmg 

4220 Advisoty and Analytic Assistance 
4230 Traatmsnt. Care, and Referral Services 
4240 Provision of Other Services 

4300 Other Resource and Service Provision Outcomes 



6000 OTHER MAINTENANCE ANO CHANGE OUTCOMES 
StOO Aesthetic- Cultural Activities, Traditions, and Conditions 
52Cv Organizational Format, Activity, and Operetion 
5300 Other Memtenance and Change 
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a Reprinted from Lennlng, Lee, Micek, and Service (1977/, page 11. The fourth-level 
Categories, Into which any of the categories listed here can be divided, are 
"maintenance 11 (a fourth digit of "1") and "change 11 (a fourth digit of "2"). 
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Figure 3 

ONE POSSIBLE SET OF STUDENT OUTCOME 
CATEGORIES FOR THE TIME DIMENSION a 



10. Short-Duration Outcomes 


>1. Short-duration 


12. Short-duration 


outcomes appear- 


outcomes appear- 


ing at or prior to 


ing after gradu- 


graduation 


ation 



20. Long-Duration Outcomes 


21. Long-duration 


22. Long-duration 


outcomes appear- 


outcomes appear- 


ing at or prior 


ing after gradu- 


to graduation 


ation 



a Reprinted from Lenning, Lee, M^cek, and Service (1977), page 29 




Figure 4 



TAXONOMY OF TIMES TO CONSIDER FOR COLLECTING UNDERGRADUATE 
STUDENT OUTCOMES DATA USING SURVEY QUESTIONNAIRES* 



100 Data from Lower Division Students 
110 Data from Freshmen Students 

111 Oata collected Prior to Fall Registration 

112 Data collected One Month Followina the First Day of Classes 

1n the Fall 

113 Oata collected One Month Prior to First Semester Final Cxams 

114 Data collected One Month After Spring Semester Classes Begin 

115 Data collected One Month Prior to Spring Semester Final £xams 

116 Other. For Example, In the Middle of the Summer Term 

120 Data From Sophomore Students 

121 Data Collected Prior to Fall Registration 

122 Data Collected One Month Following the First Day of Classes 

1n the Fall 

123 Data Collected One Month Prior to First Semester Final E<ams 

124 Oata Collected One Month After Spring Semester Classes Begin 

125 Data Collected One Month Prior to Spring Semester Final Exams 

126 Other, For Example, In the Middle of the Summer Term 

200 Data From Upper 01 vision Students 
210 Data From Junior Students 

211 Data collected Prior to Fall Registration 

212 Oata collected One Month Following the First Day of Classes 

1,< the Fall 

213 Data collected One Month Prior to First Semester Final Pxams 

214 Data collected One Month After Spring Semester Classes Begin 

215 Data collected One Month Prior to Spring Semester F1na* Exams 

216 Other, For Example, In the Middle of the Summer Term 

220 Data From Senior Students 

221 Data Collected Prior to Fall Registration 

222 Data Collected One Month Following the First Day of Classes 

in the Fall 

223 Oata Collected One Montii Prior to First Semester F'nal Exams 

224 Oata Collected One Month After Spring Semester Classes Begin 

225 Oata Collected One Month Prior to Spring tester Final Exams 

226 Other, For Example, In the Middle of the summer Term 

300 Data from Alumni 

310 Data Collected at Graduation 

3i :) Data Collected One Year After Graduation 

330 Data Collected Five Years After Graduation 

340 Other, For Example, Data Collected 20 Years After Graduation 

400 Other, For Example, Data Collected After Students Have "Dropped Out" 



Reprinted from Lenning (1977), page 8. These categories and subcategories 
are based specially on the data collection experiences of staff in the Office of 
the Vice Chancellor for Acauenlc Affairs at the University of Colorado, Boulder. 
Therefore, they may not be entirely appropriate for other postsecondary insti- 
tutions. 



When the three dimensions are put together, they can be pictured graph 
ically as a three dimensional series of cubes formed by the categories for 
one of the dimension intersecting each of the categories of the other two 
dimensions. This is illustrated below for the broadest categories of each 
dimension. 



Individual/ 
6roup Clients 



Interest- 
Based Clients 

E 

*-* 

VI 

% Geographic- 
° Based Clients 



§ Aggregates 
5 of People 



Other 
Audiences 




TYPE-OF-OUTCOf-IE DUOS ION 




It is illustrated at a more detailed outcome category level as follows, 
using the category code numbers shown in Figures 1-3: 



AUDIENCE 
12—Family and Relatives of 
Students and Former 
Students 



TYPE-OF-OUTCOrX 
1132— Changa in Income and 
Standard of Living 



TIME 

22--Appearing after graduar 
tion and having long' 
duration 



r 

12.1132.22 
f 



An outcome for the 
family and relatives 
of students and fom-sr 
students that involves 
/ a change i" incon-3 ana 
standard of living ih&z 
will appear after 
graduation and oc of 
long duration. 



When the dimensions are combined at the level of detail Illustrated above, 
they provide several thousand distinct "cells". The categories within the 
Structure are believed to cover the full range of possible audiences and types 
of outcomes, and procedures are provided for subdividing to even more detailed 
subcategories than provided by the Structure. Thus, when all of the outcomes 
1n a list are categorized using the Structure and step-by-step procedures for 
this purpose, areas of the Structure that have no outcomes assigned to them 
(or fewer than expecteu or wanted) become readily apparent. Then, 1f desired, 
specific priority outcomes for those areas that seem lacking can be generated 
using another set of procedures that have been developed. This, in summary, 
is the process that was used 1n this tryout of the Structure. 
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THE UNIVERSITY OF COLORADO OUTCOMES LIST AND ITS DEVELOPMENT 

The Student Outcomes Planning Model helped guide the development of the 
outcomes list used by the University of Colorado at Boulder. This model is 
based on the notion that information about students is useful in determining 
educational policy and improving curriculum. The Model allows for the study 
of various student subpopulations and their differential outcomes. It repre- 
sents an exploratory investigation into types of relationships between certain 
student characteristics and outcomes variables. The Model evolved at the insti- 
tution in 1975 to assist internal manangement, to provide information for exter- 
nal accountability, and academic planning. Academic planning Includes needs 
assessment, program development, evaluation, and resource allocation (budgeting;. 

The Model allows for the systematic collection of student data through 
the use of surveys. The Model facilitates the coordination of basic student- 
related questionnaires and incorporates the results into a larger, more exten- 
sive academic planning model (Meyerson and Banfleld, 1955). uther important 
items of outcomes Information which are available through institutional records 
(for example, college entrance exam scores and undergraduate grade point aver- 
age) are not Included. The Model 1s designed to identify those items that 
must be collected directly from students. 

The Student Outcomes Planning Model includes students' background, atti- 
tudes, values, aspirations, abilities, and the extent of change 1n these var- 
iables. The Model, ?hown in Figure 5, consists of four basic questionnaires: 
1) Freshman Questionnaire, 2) Exiting Students' Survey, 3) Graduating Students' 
Questionnaire, and 4) Alumni Survey. The Model allows for longitudinal (cohort) 
and cross-sectional analysis. 



Developed by Richard L. Harpel and Jean J. Endo 
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STUDENT OUTCOMES PLANNING MODEL 
FOR 

THE UNIVERSITY OF COLORADO— BOULDER 



cn 

CD 



FRESHMEN 



DEMOGRAPHIC CHARACTERISTICS 



Academic abi I ity 
Financial support 



PREVIOUS EDUCATIONAL EXPERIENCES 



ASPIRATIONS/EXPECTATIONS/ 
MOTIVATIONS 



Reasons for choosing CU 
Highest degree planned 
Career plans 



SELF EVALUATION 



Academic 
Social 



CRITICAL THINKING ORIENTATION 



COALS/VALUES 



EXITING STUDENTS 



TIME OF DEPARTURE 




REASONS FOR LEAVING 




FINANCIAL SUPPORT 




EVALUATION OF UNIVERSITY 


ENVIRONMENT 





Academic 
Social 



UNMET NEEDS 



men 



FUTURE PLANS 



GRADUATING STUDENTS 

[ACAD EMIC ACHIEVEMENT 



Grade point average 
Degree type 
Basic academic skills 
Critical thinking 

orientation 
Satisfaction with college 

experience 



CAREER DEVELOPMENT 



Major or field of study 
Highest degree planned 
Importance of job factors 
Satisfaction with college 
experience 



PERSONAL/SOCIAL DEVELOPMENT 



Interpersonal skills 
Personal talents/creativity 
Appreciation for culture 
Understanding different 
cultures/ideas 



EVALUATION OF UNIVERSITY 
ENVIRONMENT 



Academic 
Social 



ALUMNI 



OCCUPATIONAL SUCCESS 



Positions held 
Skills required 
Assessment of ski lis 
Job characteristics 
Flexibility in field 

changes 
Satisfaction with 

job 

Satisfaction with 
college preparation 



FURTHER ACADEMIC 
EXPERIENCES 



Graduate School 
Continuing education 
Military occupational 
training 



COMMUNITY ACTIVITY 



CI ubs/organ i 2a t ions 



| POL 



POLITICAL PARTICIPATION 



Elections 
National events/ 
local activities 



CULTURAL INTEREST 



Level of participation 
Breadth of Interest 
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Longitudinal studies involve testing the same students (cohorts) several 
times in their college career. Questionnaires are given to students as fresh- 
men, exitinq students, graduating students, and aiumni. ihe testing of cohorts 
controls for many external variables such as family background and basic ability. 
Unfortunately, much needed and useful data is not produced early 1n the process. 

Cross-sectional studies provide responses from a sample of students 
representative of those 1n the longitudinal cohorts. Cross -sectional studies 
are performed on exiting students, graduating students, and alumni. 

The Model 1s currently in Its third year of testing— Incorporating both 
the longitudinal and cross-sectional surveys to determine response differences. 
It 1s hoped that within two years the University I be able to minimize the 
length of the surveys and use only those Instruments needed to provide student 
data necessary for efficient planning and management of campus programs. 

The Student Outcomes Planning Model centers around 22 student related 
goals and their measures which are listed in the Freshman Questionnaire. The 
relative Importance of each goal reflects students' expectations while attending 
the University. Student Interests are an important component of educational 
planning. Administrators can balance tne needs of various student populations 
and the different preferences of various educational experts. Several measur- 
able objectives from the Higher Education Measures ar.d Evaluation KIT {C. Robert 
Pace, 1975) were assigned to each goal. For example, the goal "to gain self- 
confidence" 1s measured by responses to the following statements: a) "I am a 
person of worth and on an equal plane with others", b) "My confidence in myself 
1s strong enough so that 1t doesn't bother me 1f people don't like me", c) "I 
seem to have inner strength in handling tilings". 
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In addition to the Pace and Associates KIT, a number of other sources 

2 

in the literature were referred to in devloping the Model and its assoc- 
iated questionnaires. To date, the Boulder campus has comoleted a 1975 
Freshman Questionnaire and a 1977 Freshman Questionnaire and is currently 
developing an Exiting Students' Survey to be completed in 1978. The fresh- 
man surveys will be administered every two years to provide descriptive infor- 
mation and establish trends. Trends in measures of objectives are useful in 
determining whether an outcome is in the desired direction. The exiting 
survey will comoare students who have temporarily or permanently left the 
institution with those who have regained. It will identify the stopouts, 
transf erouts , and dropouts. It will determine why students leave the campus, 
the time they left, amount of financial support received, and what their 
future plans wMl be. 

The Graduating Students' Survey will be administered 1n 1 979. It will 
include questions that attempt to measure academic achievement, career develop- 
ment, personal/social development, and an evaluation of the University's environ- 
ment. One year later, the alumni survey will measure occupational success, 
further academic experiences, community activities, political participation, 
and cultural Interests. 

PROCEDURE USED BY THE UNIVERSITY IN THE CLASSIFICATION OF THE ITEMS ON ITS 
OUTCOMES List 

The University of Colorado outcomes 11st was derived 1n part from the 
Student Outcomes Planning Model, *no 1n part from other sources. Although 
the primary focus of the 11st 1« on student outcomes, a few outcomes for 
other audiences are also included, that are not student outcomes but might 

7 

Additional sources used 1n developing the Model were Schalman, et al . 
(1974; , Balrd (1967), Baldrldge (1971), Bloom (1956), Clapp (1946), Cohen and 
and March (1974), Fox (1974), Peterson (1973), Richman and Farmer (1974), 
Trfvette (1973), Meyerson and Banfleld (1955), and M1cek and Arney ( 1 974) . 
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be expected to relate to student outcomes, for example, number of publications 
I by faculty, number of tjoks 1~ the library, and number of events and type spon- 
sored by the departments or universities. 
I Two of the basic processes developed by the NCHEMS staff for using tne 

■ Structure were involved in this study. Systematic, straight forward, step-by- 
step procedures for each of the orocesses are provided in Lenn^g ( 1 977) . First, 

I process No. 2— classifying outcomes items -was used, to classify all items in 
— the University of Colorado outcomes 11st. Then Process No. I was applied in 

■ order to evaluate tne adequacy of the list, in terms of wnetner Important out- 
I comes desired by the University community have been left out. 

I PROCESS NO. 2— CLASSIFYING THE OUTCOMES ON THE LIST 

_ The desired "audience" for classification was determined to be current and 

graduating students (Code No. 11). First, each outcome on the University of 

I Colorado list was classified 1n terms of tne broadest type-of-outcome categor- 

ies. Then each was grouped into tne more detailed categories at the second and 

I third levels of detail for that dimension of the Structure. Procedures are 

■ provided for adding additional levels of detail, using available taxonomies tnat 
go into still more detail, for example, Bloom's {19$6) taxonomy of the cognitive 

I domain, or using a logical array developed locally. At this University it 

decided that the third-level Structure categories were adequate for their aca- 

■ demic planning purposes. 

I Table 1 shows tne University of Colorado's listma of outcome items (rignt 

half of each page) and tne Structure categories that each was classified into 
I (left half of each pace). It was noticed tnat a number of Structure categories 

■ did not have outcomes list Items assiqned to them. It was felt that there 
might be additional outcomes categories to which some outcomes from tne list 

I nad been assigned so Process No. 1 Procedures followed. 

Mc 61 64 



Table 1 



NCHEMS 
Category 
Code 
Number 



1130 



2110 



2210 



2230 



2250 



2270 



ERIC 



-:io 

2420 

2620 
2680 

2710 
2740 



THE OUTCOMES LIST ITEMS PLUS THE 
STRUCTURE CATEGORY INTO WHICH EACH WAS PLACED 



11 AUMENCE" --SENIOR STUDENTS AT CU 

NCHEMS 
Type-of-Outcome 
Category 
Name 

ECONOMIC OUTCOME (Code 1000) CATEGORIES^ 



Items Included 
on the 
University of Colorado 
Outcomes List 



Income and Standard of Living 



- Family income 



HUMAN CHARACTERISTIC (Code 2000) CATEGORIES 



Desires , Aims, Goals 



Academic Skills 



Creativity Skills 



Intellectual Skills 



Occupational Skills 



Attitudes, Values 
Perception of Self 

Autonomy and Independence 
Tolerance and Persistence 

Completion or Achievement Award 
Licensing and Certification 
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Change in students' goals, 
desires, aspirations as 
a result of college 

Grades earned by students 

Persistence in college 

Self report of ability in 
math, writing, reading 
and comprehension 

Changes in test score that 
measure originality and 
creative ability 

Self-report of development 
and activity 

Change in students' abilit) 
to analyze or solve prob« 
lems 

Measure critical thinking 
activity by developing 
an "index score" 

Demonstrated ability to 
perform specific tasks 

Self-report of occupationa 
skills 

Effect of college on 
attitudes and values 

Self-confidence measure 

Expectations 

Measures of independence 

Measures of tolerance and 
persistence 

Graduation diploma 

Special awards 

Percent passed specific 
licensing exams 



2750 



2840 



Table 1 (continued) 

HUMAN CHARACTERISTIC CATEGORIES (continued) 

Obtaininq a Job or Admission to a 
Follow-up Program 

Career and Vocational Roles 



Percent who received jobs 

Percent who were accepted 
to graduate programs 

Self-report 



KNOWLEDGE, TECHNOLOGY, AND ART FORM OUTCOME (Code 3000) CATEGORIES 



3110 



3120 



3210 



3220 



3310 



3400 



Knowledge and Understanding of 
General Facts and Terminology 



Knowledge and Understanding of 
General Processes 



Knowledge and Understanding of Spec- 
ialized Facts and Terminology 

Knowledge and Understanding of Spec- 
ialized Processes 

Research and Scholarship Knowledge 
and Understanding 



Art Forms and Works 



CLEP exam score 

Graduate Record Exam score 

Self-report 

Comprehension of general 
processes and methods 

Student grades in a general 
survey course 

Scores on tests that measur 
knowledge in specific 
fields 

Scores on tests that mea- 
sure knowledge in specif i 
fields 

Number of publications by 
students 

Number of publications by 
faculty 

Specific accomplishments 
in the arts by discipline 
or department 



RESOURCE AND SERVICE PROVISION OUTCOME (Code 4000) CATEGORIES 



4110 



Provision of Facilities 



4120 

4210 
4220 
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4230 



Provision or Sponsorship of Events 
Teaching 

Advisory and Analytic Assistance 

Treatment, Care, and Referral Services 
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- Number of books in library 

- Number of courts in the 

Recreation Center 

- Number and type of compute! 

- Number of keypunch machine! 

- Number of events and type 

sponsored by departments 
or unwersity 

- Handbook on evaluations of 

specific instructors 

- Number of advisors avail- 

able to students 

- Technical assistance avail 

able to students 

- Student health center 

- Mental health facility 

- Personal counsel fng 



PROCESS NO. I— •DEVELOPING OUTCOMES LISTS 

Process 1 was used to develop a list of imDOrtant outcomes irrespective 
of the current 11st. Such a list will vary depending upon the values of the 
planning group making the decision: for example, faculty, students, adminis- 
trators, legislators. However, an overall Institution-wide approach was taken. 

The "audience" focus was limited to "current students 1n order to limit 
the scope of Process 1. The five major categories of the type-of -outcome 
dimension were outlined on a sheet of paper that was titled "audience— currently 
enrolled students". Then, under each major category, all subcategories at the 
second and third levels of detail were considered for importance. The detailed 
subcategories were then used to stimulate thinking about specific outcomes 
that could be considered "essential" or "Important". Aiding in this process 
for each subcategory were: (1) A product/event/condition typology, (2) a main- 
tenance/cbance typology, (3) an output/impact typology, and (4) and an intended- 
un1ntended/valued-not valued typology. 

Table 2 presents Structure categories left out of the original University 
of Colorado listing that were Identified during this project as Important 
categories for the University and Its students. Also Included are the specific 
priority outcomes that were identified for each category. 
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Table 2 



IMPORTANT STUDENT OUTCOMES ITEMS IDENTIFIED 
THAT WERE NOT ON THE ORIGINAL* 
UNIVERSITY OF COLORADO LIST 



NCHEMS 
Category 
Cole 
Number 



NCHEMS 
Type-of-Outcome 
Category 
Name 



Items to be Added 
to the 
University of Colorado 
Outcomes List 



1120 
1130 



ECONOMIC OUTCOME (Code 1000) CATEGORIES 



Economic Flexibility and Independence 
Standard of Living 



- Social Mobility 

- Fam1 ly assests 

(other than income) 



2240 



HUMAN CHARACTERISTIC OUTCOME (Code 2000) CATEGORIES 
Expression and Communication Skills 



Self -perception of 
skills 1n the for- 
eign languages 



2260 



2630 



2650 



1330 



Interpersonal* Leadership and. 
Organization Skills 



Dependability and Responsibility 



Flexibility and Adaptability 



KNOWLEDGE, TECHNOLOGY, AND ART FORM 
OUTCOME (Code 3000) CATEGORIES 



Knowledge and Understanding of General 
Theory 



Self -perception of 
Interpersonal and 
leadership skills 

Positions held 1n 
organizations that 
require leadership 

Self -perception of 
dependability and 
responsibility 

Employer's opinion 

Self -perception of 
adaptabl 1 1 ty 
(other than 1n job 
changes) 



Score on tests measur- 
ing comprehension 
of general theories 



3230 
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Knowledge and Understanding of 
Specialized Theory 
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Scores on tests measur 
1ng theoretical 
knowledge 1n specif 
fields 



CONCLUSION 

As a result of this tryout of the NCHtMS Outcomes Structure, some impor- 
tant outcomes information were discovered to have been omitted in the original 
11st. The Structure served as a reminder of the breadth of outcome items assoc- 
iated with each outcome category. For example, the category "Competence and 
Skills" indicated tha*- Expression and Communication Skills such as skills in 
the foreign languages were -mportant indicators of entering students' ability. 
Basic speaking, writing, and reading skills in the English language were con- 
sidered to be the primary indicators in the orginal outcomes list. *.s a result, 
the list and the Freshman questionnaire were modified. 

The final distribution code numbers indicated that the Boulder Campus out- 
comes list is predominately comprised of human characteristics. This is be- 
cause the variables in the Model are primarily "student-oriented" outcomes. 

In addition, the classification of unique "audiences" will be valuable to 
the Boulder Campus as the outcomes information 1s incorporated into reports. 
A public relations staff can categorize outcomes items that would be useful to 
particular audiences. For example, research projects completed by graduate 
students may be of interest to the following audiences: 1) current students, 
2) public school aistricts, 3) industry and business, 4) citizens and policy- 
makers of Colorado. Short executive summaries written for particular audiences 
can be developed which communicate? this information. 

The organization which results from the classification into "type of 
outcome" can help campus leaders and representatives report the outcomes of 
educational programs more effectively. Information can be stored in computer- 
ized student information systems keyed to the NChEMS classification codes. 
The classification system can also be used to organize reference materials 
collected on outcome variables. 
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FR*! I'M. NARY TRYOUT OF THE NCHEMS OUTCOMES STRUCTURE AT TWO SETS OF FOUR COLLEGES 



Oscar T. Lenning 
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Management Systems 



In this "age of accountability," institutional researchers and others- 
have become especially concerned about concretely identifying and understanding 
the imoacts of their institution on students and society. After two years of 
concentrated effort NCHEMS has developed an "outcomes structure," a new system 
for organizing outcomes and outcome information for purposes of classification, 
analysis, and decision-making. As conceived, this structure nas potential use 
in helping institutional officials to identify educational needs, develop goals, 
translate goals into more concrete objectives, evaluate tne institution and its 
programs, raise the awareness of institutional personnel about the need to 
rethink the outcomes of their institution or program, explore w» »r or not 
there are important outcome areas that have been overlooked on camous, com- 
municate about outcomes with concerned publics, etc. It does this through 
direct application to: (1) defining outcomes, (2) organizing outcome infor- 
mation, (3) generating lists of priority outcomes, (4) classifying outcomes, 
and (5) information storage and retrieval. 

This paper gives an overview of the Structure and reports on a project 
i,a attenuated, in a preliminary way, to test application of the Structure in 
small liberal arts colleges. The project was co-sponsored by the Learning 
Cutccmes Task Force of the Council for the Advancement of Small Col 1 ages and NCtcMS. 
It commenced in the summer of 1975 and was completed in lace winter * 9 77 . 

— _ . 

'A paper presented at the Annual Forum of the Association for Institutional 
Research, Montreal, Quebec, May 1977. Acknowledgement and appreciation is nereoy 
expressed to the colleges participating in tnis project, to their camous coordinators 
for the project, and to the memoers of the CASC Learning Outcomes Task Force for 
■their contribution to this project's success. 
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The NCHEMS Outcomes Structure 
1 In the early 1970 ( s NCHEMS staff had developed an Inventory of Higher 

Education Outcome Variables and Measures (Micek and Wallhaus, 1973) that 
™ consisted of organized lists of outcomes prominent in higher education, 
■ along with definitions and outcome measures or indicators for each. This 

inventory was used at a number of colleges and universities in goal setting 
| and gnal translation efforts, and found to be quite helpful. Concerns were 
m expressed, however, that it was not comprehensive enough— that it left out 
* outcomes that are important to some people in higher education. Furthermore, 
M it was felt by many that the instrument should focus on the whole of post- 
secondary education, not just on higher education ♦ Therefore, an effort 
| was begun at NCHEJ1S in 1974 to develop a comprehensive structure for the 
a outcomes of posts econdary education. 

™ Before trying to develop the Structure itself; it was felt that agree- 

I ment should be reached on "just what is an educational outcome? ' 1 Various 

people seemed to view outcomes in quite different ways, so it was felt that 
| one definition of "outcome" might not be enough for the Structure— the 
B definition might have to be adjusted for different contexts in which the 

Structure would be used. Therefore, an extensive search of the literature 
M was conducted to try to arrive at a concept for educational outcomes ucon 

whicn different people could agree, and that would be especially appropriate 
| r planning, management, and policy development purposes. Concurrent with 

_ this, other comprehensive reviews of the literature were conducted to derive 

the following: all previous attempts to structure educational outcomes and 
1 outcome-related concepts such as goals and objectives that are outlined in the 

literature, all specific and general outcomes of postseconaary education that 
| are specifically claimed in various portions of the literature to be worthy of 
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concern, which could be useo to test the concent coverage the structure to 
be developed; and all princip 1 es or criteria that could be found in the 
literature of the feld of taxonomy. 

The concept of educational outcome that was derived to undergird tne outcomes 
structure to be developed (and which is thus a part of the structure) and the 
principles or criteria for developing and testing out the structure that came 
from the taxonomic literature are discussed in depth by Lenning, Lee, Micek, and 
Service (in press). A document discussing the more than 80 previous outcomes 
classification attempts found in the literature (Lenning, in press) and a paper 
that lists all the various outcomes found to be emphasized anywhere in the 
literature (Lenning, 1976) are also available from NCHEMS. 
The Concept of an "Educational Outcome" 

During the initial phase of developing the NCHEMS Outcomes Structure, a 
general concept of "educational outcome" was derived which it was felt most 
people could support no matter what their orientation-*whether their concern 
is primarily with efficiency or whether it is primarily witn effectiveness . 
Six general attributes (or characteristics) of an educational outcome were 
formulated along with five other factors that arr- important for understanding 
what a particular educational outcome is all about. For an in-depth discussion 
of each attribute and factor, see Lenning, Lee, Micek, and Service (in press). 

The attributes of an "Educational Outcome" . 7ne six attributes of educational 
outcomes have been titled: form, change status, focus, neutrality, measurabil ity , 
and output/ impact. Each is briefly describee below. 

1. FORM - This attribute of an outcome refers to the .nakeup or substance c~ tne 
outcome, that is, the forms in which particular direct outcomes of postsacondary 
education, or consequences associated with those direct outcomes, are (or are 
intended to be) observed and/or measured. The three classes cf "form" are 
0 defined as fol lows : 
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Product— tangible, concrete entities that endure, for examole, a program completer, 
a degree, a job, or a book. 



Event— observable, tangible transactions or sets of behaviors that do not endure 
with time, such as a seminar, a concert, a graduation exercise, and being listed in 
Who's Who. 



Condition— intangible but real circumstances, such as morale, satisfaction, an 
attitude or belief, an appreciation, social equity, and achievement. 

2. CHANGE STATUS - This attribute was suggested by the extensive work of Oerr (1973), 
who developed a taxonomy of tne "social function" of education that had such a concept 
as its foundation. Two basic change states are possible: 

Maintenance— outcomes that resuR in keeping the status quo; in stabil i zation, or 
in reproduction and preservation— for example, helping a student to keep basic 
academic skills from becoming "rusty 11 or continuing traditions into the next 
generatT on . 

C hange— Outcomes that result in alteration of the status quo; in modification, 
revision (improvement or otherwise), or "placement. 

3. FOCUS - Webster's definition of "focus" is "a point to which something converges," 
and this attribute converges on the basic, specific "what" that is maintained or 
changed to constitute the outcome of concern. (Another appropriate name for tnis 
attribute would have been "aspect," as used by the Swedish LISRU taxonomy of 
educational objectives [KTingberg, 1970].) To illustrate, instruction can involve 
maintenance or change on such entities as knowledge and understanding, skills and 
competencies, attitudes and values, appreciations, habits, ^oles, refutation, 3NP , 
certification and licensure, jobs, income, family relations, social conditions, etc. 

4. NEUTRALITY - Tne generic concept of "educational outcome" is a neutral one 
separated from any inherent value status. It is important that postsecondary 
education planners and managers not let values cause them to ignore important 
negative or unexpected outcomes in their planning and assessment. 

5. MEASURABILITY - This attribute refers to the extent and ease with which a 
oarticular outcome or type of outcome can be quantified. Knowledge about measur- 
ability has important implications for outcomes identification, analysis, and 
interpretation. 
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6. OUTPUT/ IMPACT - ;, 0utput" has often been used as though synonymous or combined 



with the term "impact,' 1 and suc.i a failure *o make a distinction between these 

two important concepts reduces the ability to identify, organize, and analyze 

outcomes. Distinctions formulated for these two terms are: 

Outcomes— the direct end products, events, or conditions that result from the 
appl ication of :ne institutional or program processes to transform the various 
inputs. Examples for institutions are achievement levels, specialization of 
knowledge, degrees, program completers, publications, and cultural events. 

Impacts— the consequences of outputs and earlier impacts for particular individuals, 
communities, or things. Tney are the indirect end products of institutional, 
program, or other activities and processes . Examples of possible (not assured) 
impacts for institutions include a program completer's increased ability to obtain 
and hold a job, the security and income or prestige that job gives the person, the 
increased gross national product that results from increased income of individuals, 
the increased standard of living and quality of life in society which may be 
associated with increased gross national product, and so forth. 

Other Factors Important in Understanding Particular Educational Outcomes . 

Although they do not describe the essence of an educational outcome, like the 

six attributes do, other factors are just as important in understanding tie outcome, 
applying outcomes information to planning, management, and evaluation: 

PRODUCER/FACILITATOR - Activities, methods, processes, programs, etc. that 
cause or influence tne outcome to happen, or the conditions chat allow it 
to happen. 

AUDTENCE - The persons, groups, organizations, communities, and other entities 
tnat receive or are affected by the educational outcome(s) of concern. Not 
delineating this factor often presents one of the major difficulties in 
identifying and understanding educational outcomes. 

INTENDED/UN INTEiNDED - Unexpressed as well as expressed motives for different 
groups desiring particular outcomes are important to consider in planning and 
management. The potential for unintended outcomes (both those viewed as 
positive and those viewed as negative) should also be considered in planning. 

FUNCTIONAL AREA - The function(s) that particular outcomes can or do serve. 

TIME - The point in time when the outcome occurs and how long the outcome lasts. 
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A Description of the NCHEMS Outcomes Structure 

The NGHEMS Outcomes Structure has three formal dimensions, where an 
outcome dimension is a continuum that can be divided into segments along which 
outcomes can be placed and viewed in relation to one another. These tnree 
dimensions are: (1) Audience— the persons, groups, or entities that receive 
and/or are affected by (or that are intended to receive or be affected by) the 
outcome of concern, (2) Type-of-Outcome— whether or not the outcome involves 
a change in status (maintenance versus change) and the basic, specific entity 
that is maintained or changed, and (3) Time— the time frame in which the out- 
come occurs or is intended or expected to occur. 

The categories and subcategories (along with their associatec code number) 
o-f the audience dimension are presented in Appendix A. For some purposes at the 
institutional, system, state, or federal It/el, the amount of detail shown at 
the lowest level in appendix A will be sufficient. For many purposes, however, 
(and especially within the institution), additional levels of detail are needed. 
Helpful orocedures for adding additional levels of detail to the dimension, for 
different purposes, are provided— such a process is called 'extending the structure. 

The categories and subcategories developed for the type-of -outcome dimension 
are presented in Aopendix 3. Standard definitions along with example outcome 
measures and indicators are also available to the user of the Structure for each 
detailed category. (Work on expanding the lists of measures .or various cate- 
gories, and on synthesizing the empirical studies in the literature that relate 
to those categories, is currently underway at NCHEMS.) As with the audience 
dimension, certain applications of the Structure will call for additional levels 
of detail on the type-of-outcome dimension— for example, if it is applied to 
curricular development and planning for courses. 
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It Is intended that those dimensions and categories not of serious concern 
to a particular user of the Structure be ignored, or modified and adapted in a 
way that will better meet local needs and situations. The same is true of the 
various proposed procedures for using the Structure that are presented in a 
document especially prepared for institutional practitioners (Lenning, in press). 
Planners and managers at the system, state, or federal level should also be 
able to make use of the procedures and techniques outlined in this manual. Only 
some of the application procedures nave been tested, and those in a preliminary 
manner, but the results thus far are positive. For example, the University of 
Colorado has developed, over a period of years, lists of student outcomes that 
their staff felt confident were comprehensive for their institution. The person 
who coordinated the development of those lists was hired by NCHEMS to apply the 
Structure to the lists, using the appropriate procedures in the draft users 
manual, to see if there were any "holes." To her surprise, several of wnat 
they considered to be very important outcome categories had been left off of 
their lists. As a result, they are revising their lists accordingly, which in 
turn will result in some modifications in their freshman student survey form. 

Only time and widespread use in a variety of different institutional and 
other settings will tell whether the supposed potential of the Structure will 
really "bear fruit." Such use will also probably suggest modifications for 
future versions of the Structure, and adaptations in procedures that need to 
be made for different types of users and for different types of institutions. 

Trying Out the Outcomes Structure 
Two approaches to outcomes identification and val idation, each using the 
NCHEMS Outcomes Structure in a different context, were attempted in the joint 
NCHEMS/CASC effort. Tie "NCHEMS approach" emphasized identification and 
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preferential weighting of outcomes, ordered according to "essential," "important," 
and "less than important" outcomes. This weighting then led to questions of 
institutional administrators, departmental chairpersons, faculty members, and 
students about assessment and evidence issues. FinaMy, the discussion moved 
toward the identification of student activities and experiences which would lead 
to, percipitate or culminate in the achievement of the outcomes. (A major refine- 
ment of this approach that used only the most global categories of the Structure 
was implemented into the planning cycle at one of the participating colleges; 
and was found to be quite helpful for evaluating their institutional and program 
goals.) The "CASC approach," by juxtaposition, was designed to identify and 
specify learning activities that had three traits: 

(1) the activity was of a publicly demonstrable nature; 

(2) the institution could exercise some control over the development 

and expression of the activity; and 

(3) the activity was expressed within a time frame related to the 

acquisition of skill or the attainment of an outcome. 

Once the activity was specified, the instrument elicited resoonses which satisfied 
the criteria question and identified th.- outcomes sought through participation 
in the activity. 

Before commenting on the relative merits of, and reactions to, the aDproac.hes, 
we would like to address the problems or needs which suggested these approaches. 
The problems are the global nature of mission/goals statements, the translation 
of goals to operational objectives, and the need for determining a connection 
between what happens on campus and the raison d'etre of the institution. Out- 
comes identification is an attempt to hypothesize and realize causal linkage 
between what a campus expostulates as its mission and those changed behaviors 
which the college can be held accountable for Our task force perceived a void 
between the all-embracing goal statements f-md in college catalogs and the 
methods of assessing student achievement— most notable of which are paper and 
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pencil exercises. Hence, the pivotal issue which our task force raised with 
faculty members is illustrated by questions such as, "How do you determine if 
a student's critical thinking capability has been enhanced as a result of * 
knowing the content of your course?" At the departmental and college-wide 
levels, similar discussion took place to determine the activities wnich were 
perceived to bring about the attainment of intended outcomes. 
The Central Metaphor 

As we were participating in the ca^e exercise at one of the colleges, a 
faculty member raised an objection to the metaphor which encompassed our pro- 
ceedings. His objection was aimed at the metaphor of a microscope, which 
attempted to isolate and identify rich and diverse activities of an educational 
program. This remark is also reminiscent of Kenneth Soulding's statement that 
measurement of teaching effectiveness is essentially "measurement of raoport." 
Furthermore, a faculty member at another institution objected to a mentality 
which did not value the spontaneous, creative, and personal interaction between 
faculty and students which cannot be predetermined or assessed. Our approaches— 
both CASC and NCHEMS— have been sensitive to these concerns and have attamotad 
to integrate the greatest amount of latitude in areas of activity statements, 
evidence of achievement, and intended outcomes. While the taaching enterprise 
is personal, spontaneous, creative, and idiosyncratic, it is also 
intentional intellectual activity. Our ap -oach has been to document what is 
intentional and to recognize our limitations regarding the spontaneous energies 
released in the educational setting. 

In addition to the metaphor of the microscope is the larger question of 
whether colleges can be held accountable for attaining student learning out- 
comes. At issue are the dynamics of governanca, pedagogical intervention and 
value added learning. Governance issues emerge around the question of goal 
setting and institutional accountability. In other words, must a Xanadu College 
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graduate exhibit attainment of specified outcomes? Issues of pedagogical 
intervention emerge around student choice and the rites of passage as specified 
in the general educational, departmental majors, and college-wide retirements. 
The retreat from requirements spawned in the '60s is oeing reassessed toward the 
chary reinstatement of some common leaning experiences. Finally, the issue of 
value added learning is central to the identification and validation of student 
learning outcomes. Failing accurate and comprehensive assessment of stuaent 
capabilities upon entry, colleges cannot expect to take credit for the promulga- 
tion of certain student learning outcomes. Moreover, in the absence zf inout 
data on student capabilities, tne assessment of specific activities wnich lead 
to outcomes becomes academic at best. 
Results of the "NCHEWS Approach" 

Since diversity in intended outcomes was desired, it was decided to invite 
small colleges to participate that were diverse on geographic distribution, size, 
institutional control, nature of the student population (residential versus 
commuter and coeducational versus single sex), curriculum, and planning/management 
orientation. Other important selection factors were the insti' 'onal 'ntarast 
in the project and willingness to participate. Once the institutions were 
identified, each appointed a campus coordinator for the project. Those recre- 
sentatives traveled to NCHEMS for an orientation and oilot implementation session. 
Participation in this stage signified the institution's commitment to tne process 
and, also, familiarized the institutional representatives with the background and 
rationale for outcomes identification and validation. A vital link was established, 
thei,, between the outside group and the institution, thereby legitimizing the 
endeavor. 

The campus coordinators organized CASC/NCHEMS visits to their collages, convening 
participants and arranging schedules for individual appointments. A workshop was 
held at the beginning of the proceedings to familiarize campus representatives with 
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the purpose and procedures for the interaction. Essentially, three activities 
were scheduled for the workshop: 

(1) introduction of CASC and OEMS representatives and the outcomes project; 

(2) orientation of participants to terminology, definitions and use of 
materials for outcomes planning at the department level; and 

(3) participation in role-playing about the use of outcomes concepts as a 
basis for allocation of funds at the institution-wide level. 

After the role-playing exercise, the CASC representative offered observations 

and the participants were asked to comment on the exercise and materials provided. 

Individuals were interviewed during the next one and one-half days by Lenning 

of NCHEMS and the CASC representative vising on camous. Interviews followed 

closely an interview in r 'ment that has been appended to this paper (see Apoendix 

C). Faculty members, students, and administrative personnel were asked to comment 

on institutional outcomes but, more importantly, to concentrate on student learning 

outcomes that they desire (or in *nd/plan to bring about) for students in their 

program areas. 

Priorities that different groups gave to the various Outcome Structure 
"type-of-outcome" dimension categories at each college are presented in Table 1 — 
because of the small numbers of respondents, percentages could not be used. 
These data are discussed in depth in Lenning (1977) along with: SDecific outcomes 
suggested as especially important for various priority categories, concrete trea- 
sures and indicators which can provide adequate avicsnca that the outcomes of 
concern have occurred, and the student activities and experiences that the various 
respondents felt would most contribute to bringing about the particular outcomes 
they had emphasized as being especially important. Although a number of note- 
worthy variations in patterns for different institutions (and groups within insti- 
tutions) are noted in Table 1; institutional diffarencas in focus and. activities 



^A survey questionnaire f orm of this instrument has since been developed 
and is currently being reviewed. 
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PRIORITY RATINGS OK OUTCOML C/iTLGOR.tS BY HIRU CLASSY OF RCSPONOKNTS 



umtuwu MIO Skills 

22 ill Aiidmlt Skills 

2220 CUucnitUt* Ami t*«lly ffcutfurtlilp Skills 
22 V) Cr*4tMly Stills 
2243 f tpitffc&iuit And CuMnuit|c«ilun Skills 
2iL3 l1.ulie4.ltM | Skill. 
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really seemed to rtand out when the emphasis shifted to a more specific, concrete 
I level. 

■ As a result of the workshop and interview activities, t.e following observa- 
tions may be made: 

I (1) concept of student learning outcomes is new (and to a degree suspect) 

on college campuses; 

I (2) role-playing the use of outcomes in the resource allocation process 

can facilitate interaction and communications; 

I (3) it is extremely difficult for most faculty to decide on outcome 

• - priorities, and to go from detailed outcome categories to specific, 

concrete outcomes, but they can do so successful Ty with tne nelp 

■ of procedures' TTfce' those de Tf heated fn this aporoa~ch; and 

(4) faculties resist the delimitations of outcomes methodology. 

I ' Tne long term Impact of the exercise is difficult t, gauge, although tne 

■ residual effect, as judged by informal comments, has been positive. For axamDle, 
many zf the faculty members outside of education exclaimed that they had never 

I tried to think about wha'. they were trying to accomplish in this way before, and 

they seemed stimulated, 
i Results of the "CASC Approach" 

■ The "CASC approach" to campus selection, preparation cf representatives 
and orientations differed markedly from the "NCHEMS approach." While selection 

I criteria for institution participation were similar, greater attention was given 

■ to minimizing the cost of project implementation. Because of limitations of 
professional staff -nd money, task force members prepared expiation materials 

I which covered the concepts, rationale and processes of the task force. Included 

in the packet sent to respondents (potential interviewees) were the foKiwing 
' papers: 

I CI) overview of the activities— outcomes concept and linkage; 

(2) introduction to the objectives of the project; 
I (3) description of procedures for project imclementation ; 



(4) an "advanced organizer" which briefly described the above three items; 

(5) selected outcomes from the NCHEMS outcomes structure; 

(6) the interview instrument (see Appendix 0). 

Two brief reactions are appropriate of this point: first, too much material was 
sent to respondents; second, too little material was read by respondents. This 
latter condition necessitated a brief orientation session during most interviews. 

Once the campuses expressed agreement to participate, explanatory materials 
were sent to each campus representative— who then distributed the materials to 
campus administrators, faculty, and students CASC representatives (in oairs) 
interviewed on campus at one hour intervals, us^ng the interview instrument 
included in Appendix D. Some of the responses to tne concepts and problems were 
admittedly negative; whether this resulted from ignorance of the material or a 
thoughtful disdain is difficult to say. Other responses were very positive, as 
if the concepts of acfvi ties-outcomes linkage oroviced a means of expressing 
educational practices in a more forthright manner. Overall, it is difficult 
to say whether the colleges benefited from the interview experience. Some of 
the factors which mitigated against the implementation were: 

(1) lack of awareness of institution as to what local benefits derive 'rom 
exercise and who sanctioned the process; 

(2) the amount of material sent to each respondent discouraged some and 
confused others; 

(3) no context had been established for tr.a interview, which meant that 
interviewers had to "make their case" to a number of interviewees; and 

(4) the mechanism for incorporating the interview findings into a planning 
process were not made clear to interviewees. 

Henc, a number of factors diminished the efficacy of the CASC approach. 3ut, 

starting with a discussion of activities, rather than outcomes, does facilitate 

early interview discussion. Faculty members especially are more reaoy to talk 

about activities tnan they are about outcomes. 
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Concl usion 

The Perceived Usefulness of the Two Approaches . Four factors are perceived 
to be important in judging these approaches: 

(1) Clarity of communications— how well were key concepts expressed and 
understood? 

(2) Perception of relevance and interrelatedness—now well did respondents 
incorporate the concept and terminology into institutional possibilities? 

(3) Institutional response— did aeds ion-makers advance the purposes of 
outcomes identification and validation? 

U) Receptivity and/or resistance— to what degree did respondents contradict 
or complement the concepts presented in the interview or in other .ormats? 

Clearly, there were both positive and negative aspects to each aoproach as 
suggested by evaluative statements reported in the previous section. Some of tne 
problems with each approach could probably have been worked out with improved inter- 
viewee preorientation, extra practice in interviewing techniques and aporoach prior 
to starting the interviews, more carefully setting the stage within the interview, 
and so forth. Even so, however, a combination of the two aporoaches may be desir- 
aole, involving: (1) carefully planned orientation workshops; (2) starting the 
interviews with activities and going to broad outcomes, as in the "CASC approach;" 
and (3) going to specific outcomes and then back again to activities, as in the 
"NCHEMS approach." 

What, then, are the benefits of the concepts, approaches, and instrumentation 
as developed by the CASC/NCHEMS effort? Two seem especially apparent. The first 
is bridge-building *n the academe and institutional planning cycle. Academic 
planners ar. confronted with a unoply of expectations and severe constraints on 
resources. Expectations either become canonized in the college's rission statements 
or they submerge, awaiting resuscitation tnrough the next foundation grant. Granting 
consensus on goals, the planners are then confront with programmatic reinforcement 
of, or deviance from, goal attainment. A second benefit, that we as researchers ha*e 
witnessed, is the stimulation of faculty members toward consideration of larger-than- 
classroom issues. At one college a professor of chemistry averred that the issues 
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which transcend courses, classes, and departments are at the heart of liberal 
education. Faculty members who reacted negatively to the concepts of outcomes 
identification were similarly stimulated to look beyond the shibboleths and pat 
assumptions that most of us shield ourselves with. In a sense, outcomes 
identification brings assumptions about procedures and commitment to purposes 
out into the open for discussion, debate, and dialogue. Sharing of goals and 
outcomes among departmental ^acuity members can be the first step toward 
collegia! relations among all faculty members. The prospect of community 
ouilding in academic environments can enrich all sectors of higher education, 
from students seeking guidance and instruction, to orofessors and administrators 
seeking a wholeness and uni :y in the academic enterprise. 

The Perceived Usefulness of the Outcomes Structure . The "MCHEMS aoproach" 
was built around the Outcomes Structure, and it did give structure and an impetus 
to the interviews. However, the >.ode numbers were disconcerting to some, as was 
tne "psychological jargon" to a ?ew others. And it was not until the ;nte<-view si 
ation that most interviewees seemed to start sensing real potential significance 
for the Structure. At the end of each interview, if there was time, the inter- 
viewee was asked to rate the potential usefulness of the Outcomes Structure, 
pased on their limited experience with it >-:ius far. Table 2 oresents tuula- 
tions of the replies to this question. These Mmited Sata indicate that the 
reactions of most respondents to the Structure was ay this time fai>ly oositive. 
Appendix £ lists the cautions and potential p;-oblems with the Structure mentioned 
oy particular respondents, and specific uses for the Structure that different 
people perceived. 
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TABLE 2 

EXPRESSED OPINIONS OF RESPONDENTS 
CONCERNING THE USEFULNESS OF THE STRUCTURE 3 





Ui u NQt 

Discuss It 


Ha n *i t*a 1 \/ 
Uc r 1 ni ue 1 j 

Useful 


Useful 


Not 

Useful 


Said That They Did Mot 
Understand it Enougn to 
Judge 


1 1 

Colleqe A i 












Administrators 


5 


11 


3 


3 


1 


Facul ty 


10 


13 


9 


0 


2 


Students 


i 


5 


1 


1 


0 



a Each numb'*- indicates how many people in that group (identified in the left 
margin) gave a particular response (identified by the column heading). 

The "CASC approach" was not built around the Structure, but utilized it *n the 
attempt to make the transition from a focus on activities to linking activities to 
outcome areas. In this sense it was central to understandings of the relationships 
of activities to the larger purposes toward which those activities are directed. 
Without the Outcomes Structure categories, the faculty members could have focused 
on activities without any systematic examination of the larger purposes. The 
Outcomes Structure was also noted to serve as a taxonomic device for communicating 
outcomes across disciplinary lines. 

The evidence thus suggests that the Outcomes Structure »as faci 1 native i" the 
efforts of both approaches. It is possible, nowever, that it could have had more 
of an impact if it had been applied in a different way. Furthermore, this is not 
to suggest that the Outcomes Structure is complete and of the best possible form 
and content. As mentioned in an earlier section, it is expected that the Structure 
will continue to develop and be improved as it is used in many different tontaxts 
within different cypes of institutions. 
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APPENDIX A 

CODED LISTING OF CATEGORIES AND SUBCATEGORIES FOR THE AUDIENCE DIMENSION 



10 i.idiv.*Mi;V"'..c :Hents-rhi5 category ' ■ >s -o jerscns or jrces of re« so«s .no i.e t— ct c» ie«t 
oTThTiQStsT wnuary edu cation u mt of w« arn ano/or their wilatc issociates, such js fimtN 
and relatives or peers. 

|| StudentS -IndWidudlS or groups of .-"iivtCualS -no currently are enrolled m the program, 
insti EaTi on , or system of postsecondary eoucatlo. 

)Z Pomer students -Individuals or grouos of individuals -ho 'overly -ere en-oTcd n :ne orogram. 
institution, or system of postsecondary education 

13, Family and Relatives o f Students or Pprmer Students 

U. Peers and Associates of Students or former Students 

15. Faculty 

16. Staff Other than Faculty 

17. Qth »r Individual/Grnup Clients,- -An sample -ouH be in individual wno «s ">one of the Jbcve but is 
served by an advisory service offered oy the college. 

20. interest-Based Communi ties^ -Thls category refers to large groups 'hat are identified as entities 
working toward a 3JTT^wJTted t/tt»r*st ?r -:c»to*. 

2i Private Enterprise Commm ties -Communities -here < «i*jur ouroose s '.nancUl remuneration 
and oroTuTTTor examoie. coroontTons, smM businesses, and farmers. 

22. Association r^wHunities -Cjrwungies .n.re ™*ers belong on the oasis *f affiliation rather 
than efnoioyment, such as unions and professional societies, 

23 Government C3wmjn<rgS- -CQfrTnuni ties designed to admimste- oovermn^nr regulations and servic-S . 
such as city nan % state department of education, and legislative commumt 4 es 

21 Monn overnmentH/P'ibMc Service Comm unitie s 3th »r :*jn _th C f»SM |uMO«£^«!uCJ nn_ "i* ^jf f -w-- 
rionorofit sen/ice organizations. sucn is spools. ioso«:Jls. -eU ire igenues, ym 
foundation*, colleges (oener than the colleoe omducmq the outcome), and research org'" 1 wtiqne. 

25 (nsnmron or rnst^Jticnal 'Jmt Producing t*« iV.cc--e --^e postsecondary i««t # !ut""i ar J «r 
•Vts -ttnin tnat institution mat are cerre«veo as tie producer/ facil'ta tor if the lutcometsj of 
concern 

?c t Qttier tnteres t-8ased Communu ies-An *xamo'e *ould be in \<\ hoc coalition *a5k force of '•ore- 
sentatives f>om two ur >nor* of trie itove areas 

30. Geooraoni ce ased Com munities— This category refers to large grouos defined on the oasis of functional 

31 t pca) Cw^mmunity — A township, city, county, netropol { ran jrea or other t/ce of 'o'-il'V Mvng 

particular ooundaries. it »s not lecessar-. V restricted to the i<?gai 0 - jur 'sdictio^l boundary, 
but t K e functional one in *nich the impact of the institution is 'or snould be) ^irec: 1 * md 
ohystcaUy felt. The boundaries will va~v with >he insti tutlon/orcgram ind outcome of :c* csrn 

22. The State 

33. A, Qegion —An aggregation of states or oarcs of states 
34 The nation 

35, An International Community 

36. Qt^er Geooraonic^ased Cpromjni'-'.es - -*n example -ouM oe a 'es^rci discovery :hjt affects or-man'/ 
oeoole living in tne eclaest iatuudes. or *n*«-e i: snows i 5 ot 
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Aggregates of °eopie» »niis category refers to iuocoouiations of peooie lisz-t-u :»i j . * 
rsrrt^iicr :'ir.inc armies skzi -raj irdi-it4 .w ^-rf^s, -^-ij :r but «no lo "ot 

'necessarily have a common Interest or mission, and therefore do not constitute communities. 

4|. ability level SubooptiUtions --Subpopu)ations defined according to level of abt'ity/oroficency 
on general intellectual functioning or specific skills— for examole. gifted, "yoicol, 
disadvantaged* or skilUd, semi-skilled, unskilled. 

12. Age Swtoooou Nations 

43. Educational i.evei SubpoptiUtions 

:a [flcoma levet Subpopuiationi 

AS Occupation Subcoou la tiens 

A6 . Physical CiseoiMty Condi Mon ^ubpnoui ^tions 

A7. jace Sutipopulations 

A8. Sex SuOpopulaticns 

Ad . Other $uch Aggregate s 

-~ 50 Other A u d 4 encef« «g<amo>r^ *Qu\ii ^e t^e naf-ral e^vircr*-»nt 'hat i -jniv»»-s » »/-*o-"^nrec 

^ researcn (wnicn m *urn i<u'd be e*pec:ed to "»m)cis on lud'e"' e< v.jc*i ^s indivi^tMU ^n<; 

rnif" co*w jnlcies) and poeulat'ons of anim«U Js^ch js t-e mi-ali affected e'fercj to *eeo Jeoteted 

h^K, « — i .-■« » - « - — ggj T ^jpy AVAILABLE 
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COOEO LISTING OP THE SECCNO-AftO 'HIRO-LEVEL SUBCATEGORIES 
FOR EACH FIRST-LEVEL CATEGORY OP TYPE-OF-OUTCOXE JlttHSIOH* 



Category 
Codt Nmnoer 


entity Se^ng laincained jp Changes 


1000 


£C2H0MIC OUTCOMES 




1 100 


M . - %r H T n ftlflj ndMTf" 9 QlltCBfl!#4 

CCOnOwl C *CC™» » ]i 0 I ngeywnuwiiw* ju^vwt"»» 

1110 economic Access 

1120 Economic FTiJiiLi ' ' ty • Aaaoceoilicy and Securuy 
1130 Income and Standard of Living 




1200 


Ec&nomic Resources and Cos's 

1210 Economic Zosts ind Efficiency 

1220 Economic Resources (Including •rroioyeesj 




1300 


Economic Production 

1310 Economic 3 -oduc::vKy jna Voduc^.on 
1320 Economic Servicfts 'rovided 




1400 


Other Economic Outcomes 


2000 


HUMAN 


characteristics 




2100 


Asotrarons 

2110 Oesires. Ai<TTS, and Coals 
2120 3M likes. L'.««s. jnd tncerests 
2130 Potlvarton or Orive '.ave* 
2140 Other Asoirationel Outcomes 




2200 


Comoettnct ind Skills 
2210 Academic Skills 

2220 Citizenship and Family «*moers.mo Skills 
2220 Creative ty Ski i is 
2240 £*oressicn *nd Communication Ski Us 
2250 Intellectual Ski 1 !? 

2250 Inurotrsonai *Liadersmo, jnd Organizational SkiTs 
2270 Occuoationai and Emilovaoi m r/ Skip's 
2230 a hysicai ano y ocor Skills 
2290 Othtr Skill Outcomes 




2300 


Morale. Satisfaction, and Affectr/e Characteristics 



2310 Attitudes and 'Mlurs 

2220 3eHafl. Commiwencs. an<< ?»ulosoony of L:fa 

2330 riflings and Etnotions 

2340 Mores, Customs, and Stancards zf Conduce 

2350 Othtr Affective Outcomes 

2400 Perceptual ttarsctenstics 

2410 Perceptual A*er»ness and Sensitivity 

2420 Perception a* Sal' 

2430 Perception < Otner* 

2440 Perception of Hi mas 

2450 Othtr Perceptual Outcomes 

2500 Personality and 9 tnona! Cooing Characteristics 

2510 Adventourousness and In::at've 

2520 Autonomy ind [ndtotndincs 

2530 Oootndao 1 1 1 :y i o a*soonsioi 1 1 v/ 

2540 Oogmatic/Ootn-Minotd. Author* :ar*an/C<mocrar.K 

2550 Flexibility and AoaotaoiH:*/ 

2560 Haoics 

2570 Piycnologicai Functioning 

2580 To l trine i and ?w stance 

2590 Othtr Psychological Outcomes 

2500 Physical and Physiological Characteristics 

2610 Physical Fitness and Vms 

2520 Physiological Heaitn 

2630 Othtr ohvstcal or Physiological Outcomes f 



The 'ourtn-level categories, -nco *n<cn any of :-e :attgones tste* 
here :an oe t<vided. art "■salntananca !a 'ourtfl li*!t of TJ and 'nange' 
(a fourth digit of T) 

3791 



APPENDIX C 
NCHEMS APPROACH" INTERVIEW INSTRUMENT 
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Person (s) being interviewed: 

Position: 



CASC/NCHEMS OUTCOMES PROJECT 
ON-CAMPUS INTERVIEW OUTLINE 



1. What specific audiences (as defined by the Outcomes StW-ure) an of serious 
and direct concern to your institutional unit? 



2. Concerning Student Learning Outcomes, which of the following outcome categories 
from the Outcomes Structure are of MOST concern to your institutional unit 
(turn to facie 4 on page 20 of An Overview of tne Outcomes Structure to see 
if there are Owners of special concern. J PLACE -ri '£' aErGRE THCSc THAT ARE 
ESSENTIAL, AN T BEFORE THOSE THAT ARE IMPORTANT, ANO A'i \V 3EF0RS THOSE 
HCT IMPORTANT. 

COMPETENCE ANO SKILLS 
2210 Academic Skills 

2220 Citizenship and Family Membership Skills 

2230 Creati 1 ty Ski 1 1 s 

22*0 Expression and Communication Skills 

~~~J2250 Intellectual Skills 

22S0 Interpersonal, Leadership, and Organizational Jki'ls 
2270 Occupational Skills 

2280 Physical and Motor Skills 



KNOWLEDGE AilO UNOERSTANOING 

3100 General Knowledge and Understanding 

3200 Specialized Knowledge and Understanding 



OTHER LEARNING OUTCOMES 

2110 Ceslres, Aims, and Goals 

2120 01 si ikes, Likes, and Interests 

21 30 Motivation or Crfve Level 
2310 Attitudes and Values 

2320 ieliefs, Commitments, and Philosophy of Life 

2330 Feelings and Emotions, for examole, appreciations and satisfaction 

2400 Perceptual Awareness, and Perceptions of Self, Others, and r n«n$s 

2510 Physical Fitness and Traits 

7JJ~2 2S2C Physiological Health 

2600 Psychological Adjustment Factors 

(wnich ones? ) 



OTHER 



r ® 
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For each category selected, what specific, observable outcomes are of < M QS7 
concern? (For example, judge's ratings of performance, amount of student 
involvement, or some product developed by the student.) 



For each outcome selected in 3, what ooservations or measures would constitu 
concrete evidence t.nat the particular Teaming outcome occurred? 
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5. For each outcome selected in 3, what specific student activities and 
exoeriences would you expect to especially contribute to the rea ization 
of' that outcome (for example, peer tutoring, participation In polulcai 
campaigns, or community service projects), and when should that activity or 
experience Really occur during the student's collage career in order to 
have the greatest impact? 



If you- mentioned ctl.er audiences (than students) in resccnse to I tarn I , wntcn 
outcome categories and specific outcomes would it oe MOST important zo Switrtula,3 
for each audience? 



7. Based on the experience you have had with it by now, to what extent do^ycu^ 
feel the Outcomes Structure has any potential to be -jseTUi in the smai . co.iece 
setting? 
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APPENDIX 0 



"CASC APPROACH" INTERVIEW INSTRUMENT 



Now that you have identified SLAs at each level, will you please 
tell us which outcomes they are designed to achieve and why you 
think they reach these outcomes? InvtTviwer ahould report naponaea 
on the appropriate /ow. 



Are there Activities which do not quite fit our definition? Please 
give us some examples. 



Could you share your thoughts on the promise of outcome centered 
higher education. Are outcomes generally measurable? Can abstract 
skills ever be certified in precise fashion. To what degree do 
yr • think your colleagues understand or accept the notion of speci- 
fy *» outcomes? Does the concept of "outcomes" hold promise for 
evirating education toward greater effectiveness? 
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INTERVIEW INSTRUMENT 



College: ____ T y° e °* Respondent: 



These questions are part of an effort to further our understanding of 
student outcomes in higher education. We hope to move beyond conven- 
tional wisdom 1n specifying outcomes and by relating these outcomes to 
activities in the college experience. 

The interviews ahould rnsJcs the interviewee feel comfortable by engaging 
in acme general discussion about the campus, the program that he or she 
represents, and something about hie or her experience in teaching. Record 
the field in which he or she teaohee and something about their experience 
in teaching. 



We would like to begin our discussion with a review of our definition 
of a Student Learning Activity. Could you tell us how an SLA differs 
from other activities in which the students participate? 



2. Let us discuss the three SLAs you identified on the Advanced Organizer. 
The interviewer 3hould record the examples on the nexv cages. Can 
you answer the questions raised on the Advanced Organizer? 



3. Can you give us other examples of SLAs at each of she three levels 
and tell us how the students demonstrate achievement? 
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ACTIVITY 
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APPENDIX £ 

CAUTIONS AND POTENTIAL PROBLEMS WITH THE NCHEMS OUTCOMES STRUCTURE 

AND 

PERCEIVED USES FOR THE STRUCTURE MENTIONED 3Y VARIOUS INTERVIEWEES 
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CAUTIOUS AKO IW^rUl ^n*jLL u b 4ITH '*Gil M $ OUTCOMES 

$t?uc:upx that «ek memhoneo various :*rEavt&E£s 



?»ted to get Uudtnis ind iljpm ?vci w td m outcome alannmg liso. 
** *ted to mow people low it wi I! 



This tind of 5*1 f analysis 1$ very 'moortant. aut will i: Takt my aifference on tre camous. 



HI que worn if /ou navt tnt -ignt stttfrtg. If let * norksnoo awty from -Jit caws. 
9 Oow't jst it units* you >iave> tht e*.*t ind energy :c clang,'.. 
* One needs t1*e away *rgsn Uy-to-cty pressures in arder ;o ^moientnt tins. 



Resistance co tfie aureaucrates in *mcn ioc-jnent 's •rttte*— f cant *jnt :a ca :a 
trouole if JSing ;t unlets /ou tan inow "S *J:at u 4<rict , y levant :a Ty aepartnent ino 
its uudtnts. 

* 9 otent:a11y very t!*» tartsuri"3--<afeguar2S sno'jli Oe auilt in :c :"sure •Ji.ic t?ie tie* ar?m 
is lot to* great (find aut tnt autssnes af tn* system tnat :m at icmeved :n i snort t*m*i 

* Ct -nay orwrise aeare tnan it aelfvars. 

* Is ait systt rea Py iltrarcncal — is nerarcny the aooroonate "deP 

* fee* funds for tn* ieoartawnt :o develop long :im worfcsnoos to Mcoroorate use a/ t*e Structure. 
*" tf ft is presented arooeriy, students coHd aecomt interested *n tne Structure 

* I Mke <t, Oyn*»tte* It could at sucn a long- tern tMnq, iow*ver. 
*" Difficult to rtacft consensus an $uc*i tnfngs— nteos a ;ood ltadtr 

* r ak*s coo imjcn tl«t— wiit is leaded *s funds for* ***** staff retreat m i "etreat setting. 

* More useful at ssail :oi leges titan tlsewne** aecause «t iavt ror* af i chance :o sat ind -nderrtj«c< 
tn* activities *?1cn aid to cuccares {tne ««y of outcom-tj and *t att :o -now our stutitnts y«r/ 

* Ntt* to surt jflnq yits in goal itttlnd at tit msti ^Jtion-^idt '«v«l. :* ^ou mr*. it we 
a"tp4rt!*nt (ivt< *it' ^o in ill ai ructions $o ^ac 4ntn : too I » ;ac tcqayitr it tJit mstuuticn- 
•idt^ lavtl at ;o Mr ioart *tlt itvtr ot io-a a "tcanc^It. 

* it it tasitrto tftink af outconts \n $otcif1c ind aoncrtta ttms it *.nt aasartntn: lavtl , so s.iouU 
sttrt aoolyin*? ^it Stmetyrt to ^aci! :onstnsus it cnac iavtl aafon you :ry :o -«c*i icmt ;ons*nsus 
an aoHtft-widt ^tls. 

*" ^ttd a navt :sea; faculty tnvolvawtnt it »• -itoartanntal Itvtl »f /ou art :oinq ia naKt aood jst 
of *Jir Struc^rt. 

*" *n ixttMtnt tnstrumtnt rf «to ind At systaw • nsvjc^ions jndtrstood ind car*.: caactd :n ay 
tost of snt* fatuity t-i :nt >^al? coiiaqt. "^t ar-biaw -ny • ?• m it:i:uots— t ::-or7 :ts-:osu:on 
and loyalty to tht i-ntrai stuolts and classicil structure. . n.Tt :on^ trains, jni jnig 
Structure as "Jity anv^sion ft. Snct it is itarntd ind .«Qi«MntK, howtvtr, tnt -Macnas could 
o* rtvtrst4 and it could sottd enr proctssa: af alarm. *q and Hvtiocmtnt. 

* A prod I tar with tftr Structure conctrns dlfftrtncts In atf*nftlons, and ft 'ncludts jaroantst. 

* As a studtnt mtjOHnq in ^tlfgious tducatfon C cand to think jn ror* oantraf tar-i. ^sra'or* 
ft <S Hmit1ng--le limns «y ootions. I *ould ort^tr i :cnolttal7 -.oan-trstd *ocrc*c*v 

!t ^ould at fitful to tnt txttnt Mat it sar/ts <"a^tr tnan «nsUv<»s. Hmmal autccmes jnould 
at amonasutd. Should lac nor* ^aootn atyond :*t $cactd oucconas. 

** t qutstfon fts rttl atric/, altnough it <s an assat tnat 's systamatiaad. 

** [ ittard fro« i staff watr *»no atttndad ^a «or'<:noo /as tart ay normng tnat i 'at af ;ooo ideas 
wtrt prtstnttd. out tnat tnty *trt atmaos avtny idtaliscic. 

* People nttd to $te "M connections soontr :a *1e»r an-<ioinc usks. i#e^e*i '1 i a i *etr«ac 



ittttng *orking *»tn f\ 




NOTE: Reprinted fror Lenning (1977). 





(JSCS «» ™6 S^UCTliSC " J AT VARIOUS :'irSWr«S lE»«rW ?Mcs:vt.v. 



Can u an nd -n jlmmn? aur orogra*. 
/amclt for camcuUr anangt. 

Htloi ona at s>s:e*atic :n amting about ouccsntf f?;. 

Encourages ««cnisis an aosirad goals MCrsr :-m art«v:c:ei. 

Can servs is * ic:nulus to eilik in tstci"c autcowe 

Defining pals iml objectives for tne aoMcgt, aiv's1ons, md course. 

*orces as to ba "ore systematic m jinking aoouc -nac *a sr* trying to aceanoinn 

'.fatal for sroQOSl! aaveioORenc. 

Sotting jcals *ftd abjtcwas. 

usaful "i sttt^q goais md anortiiMO -nan 2* s /alua aas-s. 

CouM ae jsetal also 'or ;erw*l fanning of college scudanta :ancamm- mat :iey -ane 
*njm cntir collie careers -31. 

Could f»elp as cavaloo utos issessctnt 'oeasts. 

[t >ias -ade m <too md Clinic idouC *nac *a art crying to acceroln". md n -a* 117 V:e 
objectives *n tat Side ?f *y "" n * conscious 

Outcomes Structure my facilitate cumcular TOrgamzation ay ulsing u tl «?•«••-• ovt-Uo 
•n contant* 

Shoulc nalo to increase sf'iciancy. 

Structure *ay 'aciiitatt operation «* i Hit cf consul of * rorcfaHo af -net ssaiWiM can 20. 
It 's systsnatic. iiWmc1us:va, adaotaoi*. md itlos :a snort tue. 
Could seaed tiie orocass af ?Unni«g md aavalccrant* 

*orca$ /ou to wink ibout alngs /ou -ouidn't ocntrwise tfiiim aocut n a concreta tanner 
u alps to soec;fy ;cals md catarmtna ar*or*tias, 
Can *elo js anearstand tHa -ny of tutcrstes. 
able to 'ocas 10m an ttt inadequacies af arograns. 

Haan : tsougnt ;n tnli «ay aafore: *ade n inert af md trowitdeaabla icout icmt of -jte tutcomei 
t us striving 'ar. 

Sfsn'ir ?n some -esoeccs to tne HOP f orm, auc ttttsr 
jiiful for dUiogu*-- for railing autst'ons. 
Cjiji arcvJdt sdeuUdon to ?uidt dirKcion. 

Could a% « an;ou# Mol for naiomg to :tctm1nt «n«c alrscdon should at aoiig- 

Ctcs «c 'Jit concrttt tsiimjs ** *«lut in aur -whinking tmi our icdons, md btUnces a** igi<nsc 
tftt oehtr. 

Coula ba awful to sradc^ts In «v«ig«ing 50 dons— «c t*it soohomors ar ^aiier tevtl , md 
csotclaUy 1 f uwi^cidsd. 

Cauid ast <C ta itlo 4 tf'nt aur 30*1?. 

Sortads tilings auC md sar/ai is a /aluaolt ahtcklist. 

This anjtdt causad na to xort w*i iaoarvnj^c auc a n«ia Sit. 

Thti la amo^bly Ocimj aone ;:.^t md s/stenatical 'v »n "ur >?«rtr^-t ^r-iry tfue't'on 
jeoiramnc), aut avrw» rjrvt if %oi»cttves -i« :o« hnrsd «*e "»av*» vjc ta.cn an jftenuaca 
leak ic rcUtion'.n'ra ar*>a«^ 'acuity /C-s^ \cvw% md wt laslr^d 2utcr,fTAa. 3~, c *<- 2 
^ar *jia Hracrjrc :oui<i «'.o<!C*aiiy :a ji*f«j * »n u»s :a<a 

* I *av« uwugnr m «i»Har -^yi arfor? and founi •! js#'«i . I ^ Mvf-| a -ctreac 'or T7 
suff :n Junior/, md <w u antid^.' , t'»o jso :f '-hit lev is- '*wc Mt. 

NOTE: Reprinted from Lenning (1977). 
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Assessment of student education il achievement has always been a 

I 

■ necessary function w.chin postsecondary education institutions, and its im- 

portance is as great today as ever. Furthermore, re-evaluation of procedures 
being used for such assessments is desirable in a day when accountability is 
a catchword, ever more diverse groups of students are entering postsecondary 
education, complaints about test bias and high attrition rates continue un- 
abated, a decrease in the size of the traditional age student pool is 
imminent, and innovative new assessment methods are being developed. The 
traditional methods of assessment are less than adequate for today's needs 
in postsecondary education, especially with regard to its use in institu- 
tional and student decision making. 

Traditionally, paper and pencil tests have been relied on almost 
exclusively in most segments of higher education for assessing educational 
progress. Not all assessment mechanisms used in traditional academic 
courses are of this variety, however. Examples of exceptions are in physical 
education, speech and drama, writing classes, art and music, professional 
internships and practicums, and many vocational-technical programs, where 
direct observation of student performance or products often constitutes 
the primary means of assessment . A wide variety of methods and approaches to 
assessing student educational progress are available. College administrators 
and faculty in all postsecondary institutions need to consider these various 
approaches — many of them practical and cost effective — as alternatives and 
supplements to methods they now rely on for assessment of student progress 
in coursework and programs, and for institution-wide assessment. Many 
college and university officials and faculty members, however, have either 
ignored or been unaware of the potential uses and benefits of such nontradi- 
tional assessment procedures. Thus, the major purpose of this issue of Research 
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Currents is to call attention to the miny alternative procedures for assessing 
student educational progress and where one can go for more information.^ 
The "What" and "Wl\y" of Such Assessment 

Traditionally, student educational progress in college has been thought 
of primarily in terms of written and oral communication competency, intellectual 
competency, and increased knowledge and understanding. It is now 
common to think in broader terms, however, to whatever developments and 
attainments are being aimed at for students in a curricular course or program 
offered by a postsecoudary education institution. Thus, Payne (1974) refers 
to both cognitive and affective outcomes of a course or program in his book on 
"thp assessment of learning". On the other hand, Ebel (1972) insists that all 
affective development involves a cognitive component, which he would contend 
is what makes it learning. Either way, educational learning could include the 
imparting of attitudes and values (Kohlberg, 1973), value clarification, con- 
fidence and self concept development, development of various personality 
characteristics, development of improved interpersonal sensitivity and func- 
tioning, development of skills ranging all of the way from learning to relax 
to complex theory and model development, or the development of occupational 
habits (such as punctuality) and skills (ranging all the way from flower 
arranging to sophisticated technological skills). As discussed here, educational 
progress is broad in concept, including the entire range of desired student 
change and attainments resulting from planned course and program activities 
occuring in the classroom, outside of the classroom, and off-campus. Of 
course, what is desirable varies from person to person; and changes considered to 
be undesirable should not be overlooked because thay can have as much or more 
consequence for students and their lives as changes considered to be desirable. 

^For an earlier but more in-depth treatment of this overall topic, see Lenning 
(1976) 
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Assessment is a term that was nude popular by psychologist Henry Murray 
and his associates in the late 1930's as they developed theories regarding 
personal need and 'methods for determining such need. It refers to the appraisal 
of persons and/or specific aspects of persons, for example, their academic 
achievements. Some assessment specialists have limited the fo^us of this 
concept to identification — have equated it to measurement. Others relate it 
also to developing understanding of the occurrence of ^rogress, and to con- 
ceptualizing recommended or alternative courses of action to foster greater 
progress that are implied by the findings — they equate it to evaluation. As 
used here, assessment is broader than measurement, but narrower in concept 
than evaluation; it involves measurement and is a component of and contributes 
to evaluation. By assessing student educational progress we mean that such 
achievement is being measured, analyzed, and appraised, and that potential 
implications of this achievement are being explored. But, alternative or 
suggested decisions regarding courses of action that may be implied are not 
being laid out during the assessment process — they are being left to the 
evaluation phase. 

Assessment of student educational progress can be applied to many 
puposes in higher education. For students, the results of such assessment 
can serve as a motivational incentive, can provide self confidence, and can 
assist self-diagnosis of strengths and weaknesses and lead to guidelines for 
self direction. For members of the college staff, the results of such 
assessment can be applied in diagnosing student problems, appraising student 
readiness, classifying students and placing them into courses and programs, 
planning and structuring student learning experiences, counseling and advising 
students, grading and awarding promotion or merit recognition, evaluating 
effectiveness and productivity, and evaluating innovations and programs. 
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Conducting A t* r essments of Educations * Pr ogres s 

Payne (1974) lists seven broad steps in assessing cognitive and affective 
learning that most assessme it specialists would agree with: (1) specifying 
detailed goals and objectives, (2) designing the assessment system, (3) 
selecting data-gathering methods, (4) collecting relevant data, (5) analyzing 
and summarizing data, (6) contrasting data and objectives, and (7) feeding 
back results. Space will only allow comments about some of them. 

As Palola and Padgett (1971) have stated, "too little attention is 
paid to defining the aims of the educational process beyond coining global 
abstractions (p. 7)." If one is to have effective guidelines for conducting 
actions to bring about accomplished ends for one's students, the course, program, 
and/or institutional goal z> that apply need to be transformed into concrete, 
observable specifics. Such concreteness is also needed for purposes of 
assessment. As found in a project conducted by the Council for the Advancement 
of Small Colleges and NCHEMS, however, this is a difficult problem for many 
people, and especially for faculty members in a number of disciplines who are 
used to thinking primarily in terms of process rather than outcomes. That 
project developed some step-by-step procedures that have been helpful (Lenning, 
1977; Lenning and Lundin, 1977), and which allow staff to start with the specific 
student learning activities taking place or with a universe of potential learning 
outcome goal categories. Magev (1972, 1975) his provided two very useful, 
interesting and easy-to-read books that outline another approach. At the program 
or institutional level, consensus rendering procedures become important for this 
process - whether it be give-and-take group discussion or more sophisticated 
approaches such as card sorts or modified Delphi techniques. 

Assessment design is another problem that is crucial to the entire effort. 
For every assessment a plan is needed that outlines the purposes of the assess- 
ment, the context in which the assessment will occur, the questions that need 
to be answered by the assessment, the data to be gathered lor answering e*ch 
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question, the data sources, data gathering and analysis strategies and pro- 
cedures, feedback procedures, costs, how everything fits together into an 
integrated rystem, etc. The design must be realistic - feasible in terms 
of f :he costs and effort required - as well as effective in providing the 
information needed to answer the pertinent decision makers 1 questions. It 
must identify and provide a rationale concerning which specific groups of 
students should be assessed, which areas of attainment and development are 
to be assessed for each group, and how the assessment procedures and strategy 
will vary by group. For example, assessing older students 1 intellectual 
progress using a standardized instrument designed for and normed on teenage 
college students would be clearly inappropriate. 

The design of an assessment for a particular course, program, or insti- 
tution should be unique and tailored specifically to the situation at hand. 
There are a variety of general models available, however, that one can modify 
to meet local needs. At Empire State College (Palola and Lehemann, 1976) they 
have daveloped a comprehensive learning assessment system that gathers data 
from student self report, instructor observations, writing samples, standardized and 

local tests, administrators, etc. Theirs is a longitudinal approach, and 
they relate 'earning outcomes to costs and a variety of other factors, plus 
suggest strategies, and procedures for sampling (when it is needed), drta 
collection, analysis and dissemination and use of the data. Their analytic 
strategy for group data involves three steps: identifying the functions of 
all variables; manipulating and reducing the raw dat* ' ^.atistics through such 
procedures as computing averages, tabulations, cross cranslations , and analysis 
of variance; and conducting multivariate analysis such as discriminant analysis. 
(Other assessment modi?^/ exc m#?e V iiiKp?e and unsophisticated analytical proce- 
dures - such as graphical profile analysis - which can oftentimes be just as 
effective.) Concerning dissemination and use of the data, they make the 
important point that the mere reporting of the data will not lead to data 
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use. For each decision maker of concern, the data should be transformed into 
a brief, succinct informational report having an appropriate format and 
content that speaks directly to his/her concerns. 

A quite different model was developed by the Division of Occupational 
Education Instruction in the New York State Education Department (1976). They 
developed useful guidelines and procedures for assessing cognitive, psychomotor, 
and affective outcomes within two different contexts: (1) when the assessment is 
to be used to assign grades and (2) when the assessment is to be used to 
evaluate and plan instruction. They concluded from their study that paper and 
pencil tests should be used for cognitive assessment in both contexts; per- 
formance tests should be used for psychomotor assessment in both contexts; 
and questionnaires, inventories, logs, diaries, and anecdotal records should 
be used for affect assessment, °?^he n area of evaluation and planning instruction. 
(Conversely, the Empire State College Model had emphasized the importance of 
having multiple types of measures to supplement one another in measuring the 
same outcome - for example, standardized tests, self report, and performance 
observations by faculty to assess reasoning ability and critical thinking.) 
Because it is difficult to obtain reliable measures of affective achievement, 
and since measures that do exist usually just assess knowledge about the attitude 
or other variable, these researchers concluded that the affective area should 
not be considered in assigning grades. 

Other different assessment models also have appeared recently. For 
example, Pottinger and Klemp (1975, 1976) developed a General Integrative 
Model that involves the use of several tests and measures for evaluating the 
integration of life skills within students, and which investigates how students 
process and integrate information — as contrasted to storage and retrieval of 
information. As another example, Hulsart (1975) applied the National Assess- 
ment of Educational Progress model to the classroom for use by the teachers 
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and students to assess educational progress. Both performance and paper and 

pencil tests are used. Still another example is a model promoted by 

Rivas (1976) for* using assessment to test hypotheses about skill development. 

Hypotheses to be tested can come from reviewing the literature, interviewing 

experts in the field, critical examination of previous assessment results, 

or qualitative (observational) field research prior to quantitative assessment. 

Over the yoars, many of the issues and problems in assessment have involved t 

measurement of change. Overcoming such problems is also a design role. Thus, in 

longitudinal assessment it is important to compare those dropping out along 

using 

the way to those remaining, / input information about the students character- 
istics and initial levels of performance or status. Even if there were no dropout 

.J . would be 

group, considering such input information / important for interpreting the 

results of the assessment of educational progress. 

Concerning another problem related to change, a number of specialists 
in the area have stated that for analysis purposes change scores or 

average change should not normally be used in the analysis. Rather, they advise 
comparing the final status of the student or group of students to the final 
status of students having similar initia, ability. For comparison across 
groups this can be accomplished by random appointment to each group initially, 
by group assignment through paired matching on input level, by comparing across 
similar initial-level strata, or through the use of sophisticated statistical 
adjustments to posttest scores that effectively equate initial levels (e.g., 
analysis of covariance) . 

If the desire is to correlate progress to other factors (in order to 
explore what the determinants of the progress might be), corrections for 
attenuation - a phenomenon that causes the observed correlations to be 
significantly lower than the true correlations - must be made. Werts and 
Linn (1976) have demonstrated in a preliminary way that a special "simplex 
model, M which allows for the measurement error that causes attenuation, works 
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well in longitudinal studies for exploring determinants of sLudonl 
academic growth. 

The Many Options J.n Choice of Measures 

One cannot have assessment without measurement of some kind, but all 
measures have weaknesses (some more so than others). Many measures, in particular 
those in the affective area, should probably be referred to as indicators so 
that their relative inaccuracy is made apparent. Thus, it is a good idea to 
use multiple measures or indicators for a particular learning outcome unless 
one has complete confidence in one of the measures. If they all indicate 
that educational progress^flas a occurred for a student, there is a greater 
probability that such progress has in fact occurred than if only one of 
the measures were used. More than one method of data collection is also 
desirable, assuming it is economically feasible, because all data collection 
methods have strengths and weaknesses that vary from method co method and 
because some work better with different kinds of groups than do others. The - 
use of multiple methods means that where one is weaker another can be stronger, 
plus the data collections system can be tailored to the characteristics of 
the diverse groups for which progress is being assessed. 

Standardized paper and pencil tests are often used in program and institu- 
tional assessments, but in spite of some demonstrated reliability and general 
validity they often do not measure specifically what is of concern in the local 
assessment for particular groups of students. Examining Buros (1972), the 
Council on Postsecondary Accreditation (1975) compendium of assessment 

instruments, and reviews in measurement journals can suggest instruments to 

, , . t M location an<i situation. It 

consider and allow one to judge their adequacy for a particular / should also be 

noted here that the ETS Undergraduate Program Area Tests have just been revised 

and updated, and two new forms of the popular Field Test in business have been 
developed . 

In many cases it will be desirable to develop local instruments tailored 
to local needs • There are an abundance of excellent books on standard measure- 
ment theory and the construction of norm-referenced tests, and the diversity 
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is such that both experienced and ine* pericnccd instrument developers can 
be served. Important new knowledge becomes available yearly in this area, 
however, so it is desirable that a person on campus who is responsible for 
consulting with faculty and staff about their assessment problems and 
procedures also be assigned to keep up-to-date on such developments. Examples 
of such noteworthy developments are: Rowley's (1974) finding that multiple 
choice tests favor certain types of students over others; Ebel's (1970) finding 
that true-false tests are preferred for some types of purposes and content; 
Anderson's (1974) finding that cloze measures are useful for certain assessment 
tasks on the local campus; Wagner's (1976) finding that there are benefits 
as well as problems with having students construct their own classroom exams; 
Wittmaier's (1976) finding that written critique "evaluations 11 placed in 
the students files instead of grades provided more incentive for his students 
to do effective work than is the case with pass/fail grading; and Shannon's 
(1975) finding, for his students, that matching tests have a special advantage 
over multiple choice tests in assessing partial knowledge, plus his finding 
that the premises in a matching test should be organized into groups of five 
— more than that number of premises per group may mean that student skills not 
of concern, such as reading comprehension, attentiveness , and rganization, 
will be tested as much as the knowledge and skills of concern. 

As illustrated by Lange, Lehmann and Mehrens (1967), revising items takes 
less time, effort, and expense than developing items from scratch. Therefore, 
sharing items among departments and institutions, and the use of storage banks 
test items that are flexible and can be easily accessed, is to be encouraged, 
as long as great care is taken to modify them appropriately for the new context 
Another potential aid consists of books containing collections of experimental 
and/or empirically tested items and scales for particular areas of concern. 
Lenning (1977) identified some available source books, and presented cautions 
regarding their use, for the following areas: attitudes, social functioning, 
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occupational attitudes and characteristics , occupational education achieve- 
ment, psychological characteristics, and self-concept. Other aelpful resources 
that should be mentioned are Micek, Service, and Young (1975) and Pace (1975). 

Locally-developed criterion-referenceo paper-and-pencil tests are 
more often of use on campuses than are norm-referenced tests. Unlike 
norm-referenced tests that attempt to rank and discriminate among students, 
criterion referenced tests focus exclusively on the level of performance or 
mastery. Among several useful books on this topic, the one by Gronlund (1973) 
on developing^ a ^ Sr °°criterion-referenced instruments is "must" reading. As 
pointed out by Airasian and Madaus (1972) , the most difficult problem in this 
kind of assessment is determining what should be the minimum, expected, and 
desired level of performance. Thus far, people have had to rely on experience, 
expert opinion, item content validity, and/or group consensus for such stan- 
dards. For examples of criterion-referenced tests, see Knapp (1974). 

As mentioned earlier, paper-and-pencil tests need to be supplemented by 
other types of data-gathering methods. This could be desirable even in areas 
such as knowledge, understanding, and analytical thinking, as demonstrated 
at Empire State College. Furthermore, in many areas paper-and-pencil tests 
are less appropriate than other methods. There are many methods for college 
administrators and faculty members to consider, and each will be useful in 
some circumstances and not in others. Fifty such assessment mechanisms were 
found mentioned in the literature, although some of them overlap, and they are 
listed below: 1. anecdotal records, 2. audio-visual media procedures, 
3. behavioral events analysis, 4. card sorts, 5. case studies, 6. charette, 

2 Each of the methods listed must be used with care, and procedures for applying 
it, plus its appropriateness for the local assessment situation, should be 
explored in depth. As an example, consider unobtrusive measures which provide 
indirect indications that a learning outcome has occured - for example, an 
increase in attendance at campus plays and art displays corresponding to the 
introduction of a course on appreciation of the arts. Thelin (1977) says 
about unobtrusive measures, "These obvious, isolated indicators have to be 
considered in clusters and tied to a conceptual framework if they are to be 
of significance for institutional monitoring [p. 113]." The reader can obtain 

(footnote continued on pageil) 
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7. checklists, 8. computer analysis, 9. computer-assisted instruction, 
10. concerns conferences, 11. confrontation exercises, 12. critical incident 
technique, 13. diaries, 14. fact finding and decision-making exercises, 
15. games (board games and otherwise), 16. group interviews, } 7- group 
problem solving exercises, 18. in-basket techniques, 19. institutional 
records and other secondary data, 20. interview stimulations, 21. inventories, 
22. job element analysis, 23. leaderless group discussions, 24. logs, 
25. on-the-job assessment by supervisors, 26. opinion polls, 27. oral exams, 
28. oral presentations, 29. organizational climate technique, 30. performance 
tests, 31. product testing and assessment, 32. programmed cases, 33. quest- 
ionnaires and rating forms, 34. ratings of expert judges or others, 35. ratings 
of self, 36. recording observed behavior or other observations, 37. reverse 
flow conferences, 38. role playing, 39. self monitoring, 40. self observation 
reports, 41. simulation games, 42. simulation models, 43. speak ups, 44. 
staffing conferences, 45. stress interviews, 46. structured interviews, 47. 
unobtrusive measures, 48. unstructured interviews, 49. work samples, 
50. written diagnostic and trouble-shooting exercises. 



(Footnote it 2 continued) an in-depth discussion of unobtrusive measures in a 
book by Webb and associates (1966). Some helpful discussions of other methods 
are also availabe, as follows: Many of the methods listed (Knapp and Sharon, 
1975); opinion polls, speak-ups, charette, group interviews, concerns conferences, 
reverse-flow conferences (Witkin, 1975); use of motion pictures, television, 
photos, kinescope, and videotape for testing (Edling, 1968); programmed cases 
and organizational climate technique (Pottinger and Klemp, 1975, 1976); quest- 
ionnaires (Oppenheim, 1966; Warren and Roeifs, 1972; Tull and Alboum, 1973; 
Bower and Renklewicz, 1977); behavioral events analysis (McClellnnd, 1974; 
Pottinger and Klemp, 1975, 1976); job elements analysis (Primoff, 1973; Pott- 
inger and Klemp, 1975, 1976); computer grading of essays (Levy and Fritz, 1972; 
Slotnick, 1972); records and other secondary data (Boyd and WestphaJl, 1972): 
self report (Baird, 1976); direct approaches to observing and recording 
behavior (Boytl and DeVault, 1966); use of self-charts of course progress 
(Swenson, 1974); objective scoring key for compositions (McGowan, 1976); 
innovative grading practices (Smith, 1976); self monitoring alternatives 
and their effect on achievement (Mount and Tirrell, 1977); and staffing con- 
ference assessment and its effect on achievement (Kelly and Dowd, 1975). 
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Conclusion 

A large number of alternative asi^ssment tools from which college ad- 
ministrators and faculty can choose h;ive been identified. Their relevance to 
what is to be assessed, and for whom, is the primary concern. 

A special word should be said about a new and growing phenomenon in 
postsecondary education— assessment of competencies pertinent to real-life 
situations. In two excellent documents (1975, 1976), Pottinger and Klemp 
present a number of competency-based measures that have been validated and 
tested in both academic and real-world settings for the following areas: cogni- 
tive outcomes, affective outcomes, and social outcomes. They also list 12 
characteristics and advantages of such measures (1976) that should be of great 
interest to many reading this article. 

The American College Testing Program has in operation an Adult Performance 
Level Program that uses a paper-and-pencil instrument to access "functional 
competency" — the ability to perform the wide variety of tasks required in modern 
society. They have also developed and pilot tested an open-ended battery using 
films, videotapes, and audio cassettes to measure the desired general education 
outcomes of communication, solving problems, and clarifying values as they apply 
to the three "real life" areas of functioning in social institutions, using 
science and technology, and using art. In addition, they have developed, but 
not yet tested, an objective paper-and-pencil version of that battery. 

Educational Testing Service has developed, but not yet validated, a paper- 
and-pencil instrument that is designed to measure competencies of college students 
in the areas of communication, analytic thinking, synthesizing ability, and social- 
cultural awareness. Students respond to open-ended questions about specific 
problem/situational cases. The instrument is easy to administer, requiring 
minimal training, and expertise. 

The CUE Center at Bowling Green State University has also developed a number 
of single-domain instruments for assessing generic competencies that are proactive, 
for example, critical thinking in managing confrontations and creating one's own 
discrimination. These instruments have been tested in a preliminary manner and 
found quite useful (see the paperback by Whimbey, 1976). 
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Introduction 



An important value that is a part of our culture is that all social 
institutions presumably exist to neet individual and group needs. This basic 
value is reflected in college student personnel work, which came into being 
as a profession solely to meet student needs. Students had needs in certain 
areas that were not being met adequately by colleges 1 instructional personnel. 
Furthermore, the instructional personnel had felt needs to devote their time 
exclusively to instruction. Thus, the tenn "extra-curricular" came into 
existence, which was largely viewed by the academic cornnunity as separate from 
and less important than formal curricular instruction. 

By the 1960*5, ther* were large numbers of college staff members primarily 
engaged in specialized services to students such as admissions, records, counseling, 
advising, orientation, housing, student out-of -class activities* learnina skills 
assistance, living-learning centers, career education, and behavioral learning. In 
addition, needs for new types of specialized services— svch as financial aid and 
honors program coordinators— were becoming more apparent. Durina the sixties, 
student personnel workers were advocatinq the position that the needs of students 
outside of the classroom were as important as the needs within the classroom, and 
that student activities and experiences within thi arena could supplement, reinforce, 
and provide a "practice area" for testing and applying classroom learnina to "real 
life" situations. Thus, the tenn "co-curricular" came into voQue within thp oro- 
fession, and even some academics came to acknowledge that the increased status 
implied by this term was warranted. 

During the 1970's, field work and other c/f-campus experiences were 
increasingly becoming an integral part of the formal curriculum, and there 
were emerginq concepts such as Internships, cooperative education, 
and external degree programs. Increased numbers of academics were also 
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starting to acknowledge the importance of influencing the affective domain as 
well as the cognitive domain with their teaching. Furthermore, increasing 
numbers of non-traditional students with unique characteristics and educational 
needs (compared with those of traditional students entering college out of high 
school) were enroll 'ng in postsecondary institutions. At the same time, many 
student affairs staff members were seeing their activities as an integral part 
of the curricular effort; with many of the out-of-class and in-class needs of 
students being very much inter-dependent. Student affairs personnel began to 
call themselves student development experts, and considered themselves a signifi- 
cant part of the total instructional team. The collaboration concept, where 
student development experts and other members of the faculty work together to 
promote student learning, has started to be viewed as the ideal goal to enhance 
total student development. (Miller & Prince, 1976). 

As part of its continuing movement towards increased status and recognition 
within the college community, the student affairs profession has acknowledged the 
importance of evaluating whether significant student needs are being met by its 
activities and programs, or whether those needs could be more effectively met 
if modifications are made. Thus, during the seventies there has been increased 
emphasis within the profession regarding the importance of program evaluation. 
The need for more sophisticated mechanisims designed to provide more objective, 
systematic, and effective evaluation procedures is being advocated frequently 
by small -col lege and university staffs. 

The evaluation literature generally acknowledges the importance of assessing 
needs as a prelude to or a part of systematic evaluation, but the procedures for 
sud assessment are rarely explicated. It is as if needs assessment is an obvious 
and elementary process. To illustrate the lack of emphasis on needs assessment 
in the evaluation literature, we can refer to what is probably the largest and 
most comprehensive bibliography of evaluation references ever developed. Despite 
603 entries in Bunda's (1976) Evaluation Bibliography , no mention of needs assess- 
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ment 1s made in the document. Yet, as discussed by Witkin (1975), needs assess- 
ment 1s a complex, difficult area that is largely undeveloped: 

The field is markedly and sadly lacking 1n almost any kind of research 
on the processes of needs assessment. There should be validity and 
re 1 lability studies on instruments, as well as studies of the effects 
of different assessment processes and communication strategies on the 
educational system. Longitudinal studies are needed to trace the 
impacts of needs assessment on policy making, curricular change, 
organizational structure , and student performance. Cost/benefit 
comparisons of different approaches are needed. Studies might also 
be undertaken to test hypotheses generated by theoretical models, 
such as Kaufman's unity continuum ... Few developers have shown 
how to relate such qualitative data as values, oerceptions and 
concerns to such quantitative data as test Ss.es, demographic data, 
and transiency and absentee rates. Yet until such techniques are 
widely available, educators will assign priorities and make decisions 
using one-dimension or over-simplified decision rules. [Pp. 7-8 J 

A sorcewhat sinilar statement can be made about assessing needs in the student 

affairs profession. Counselors and other student affairs staff members are generally 

quite knowledgeable and proficient in identifying and assessing important needs of 

their individual clients. Yet, when it comes to setting up programs for groups, 

the identification and assessment of group needs is often quite unobjective, 

unsystematic, overly simplistic, and often ineffective. Even when writers in 

student affairs do discuss needs assessment as a prerequisite to or part of 

evaluation, they tend to deal with it in an introductory fashion, as in Burck 

and Peterson's (1975) discussion of five recommended steps for good program 

development and evaluation. Of course, in student affairs there 1s an additional 

problem (which may also be a problem in other areas) in that little meaningful 

or systematic evaluation has been considered or attempted (Burck & Peterson, 1975). 

Thus, a separate chapter on needs assessment has been included in this 

evaluation monograph. In this chapter, the concept cf "need" will be clarified, 

and the importance of "which needs for whom" will be discussed, along with 

relevant classifications of needs that can be useful in planning student services 

needs assessment. Alternative approaches, strategies, techniques, and procedures 

having relevance for such assessment will also be examined. 
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Whose Needs? 

Before needs can be assessed, it is essential to «sk the following 
question: Which specific groups' needs is it important to identify and under- 
stand for planning and evaluating a particular student affairs program? All 
students may have some needs in common that the program could help meet (e.g., 
the need for organized group recreational activities). On the other hand, 
various subgroups of students are likely to have different types and patterns 
of needs. For example, the following special groups of students have been found 
to have special counseling and/or other student service needs: commuting 
students versus resident students (Chickering 1974); environmentally handicapped 
students (Kapel, 1971); married students (Flores, 1974); minority students 
(Moore, 1970); students such as homemakers, military veterans, retirees, and 
retrainees (Cross, 1978; Lenning & Hanson, 1977); and physically handicapped 
students (Coffing, Hodson, & Hutchinson, 1973). 

In order to effectively improve student affairs programs, it is also 
important to identify and assess the needs of particular non-student groups. 
For example, as a student affairs program planner and administrator, it is 
important to know whether one's staff members have adequate time with their 
clients; time for consultation with other staff, keeping up with the literature, 
writing case reports, doing client-centered research, reflection between 
client sessions, and for continuing education activities; adequate psychometric 
resources and other staff support; staff organization and functioning needs, 
etc. Other nonstudent groups who may have needs of concern for various types 
of student affairs programs include: the families of students, prospective 
students, high school counselors, alumni, dropouts, and the local community. 
For example, all of these groups have informational needs. 

Which groups should be of concern in assessing needs depends on the 
student affairs program context or situation. For example, the institution's 
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size, purpose and mission, goals, and constituencies all have a direct 
bearing on Identification of groups for focus. A comprehensive taxonomy of 
possible target groups in postsecondary education— that could possibly 
stimulate reflection regarding possible target groups for a needs assessment 
study—is presented in Figure 1. This taxonomy constitutes one dimension 
of the NCHEMS Outcomes Structure (Lenning, Lee, Micek, & Service, 1977). 



Figure 1 goes about here 

The importance of being concrete and specific about whose needs are of 
concern cannot be over-emphasized. For example, too many assessments of students 
needs have focused on "students in general" as their target group, with little 
or no subgrouping. (Once the specific student groups' needs have been identified 
and assessed, need similarities and discrepancies across groups can be examined 
and compared for general student needs.) In addition, it should be emphasized 
that target groups can be too specific in nature. Focusing separately on groups 
that are not that unique makes the needs assessment unduly cumbersome and may 
decrease the usefulness of the assessment data for planning purposes. 

The taxonomy in Figure 1 may suggest important target groups that would 
otherwise be overlooked. To illustrate, the category titled "association 
communities" emphasizes that we should also be concerned about the needs of 
our profession, e.g.,. with the need for sharing our innovations, experiences, 
and research with our professional collegues throughout the country. 

Clarifying the Concept of Need 
In most cases, needs assessment studies have been based on a discrepancy 
concept of need, such as the discrepancy between "what should be" and "what is 
currently the case." During the I960* s , process and procedural discrepancies 
between the actual and the ideal were generally included. However, since 
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CATEGORIES OF PERSONS, GROUPS, AND OTHER ENTITIES 
OF POSSIBLE CONCERN IN ASSESSMENTS OF NEEDS 3 



10 to*w*#*/Qf*# C/fefftt-Thte category refers to persons or grouPi of parsons who ere <tooc\ clients of (ho postsecondary education unit of 
concern end/or the* immediate eesoclatee. such es family and relatives or paart 

11. Sfuoanfs-* Individuals or groups of Individuals who currently ere enrolled in t ha program. Institution, or system of posi secondary aducaiion. 

12. Form* $»Wsnfs— individuals or groups of InOiviOuait who formerly were anroilad in the program, institution, or system of potisacondary 
aducaiion. 

13. fem*fy and flefeffres of Students or Foemm Studontt 

14. Atarsantf Atsoctarat of Sfudenff or Foemor Studomt 
tl feouffy 

16. SfttOtfmttmFmuHy 

17. Off»ar moYwouaf/Group C0ar*s— An example would be an individual *ho li none of ihe above but is served by an advisory service offered 
by the college. 

». toraraaf-iaaad Communitm—TM* eatagory refere to large groupe thai ara identified aa anmtaa working toward a weff-deffned ^farasf or 

21 *rve* *trerprts» Comrfww»mas-Cornmun4ilee where a major purpoae la financial remunerotron and profit -for esampie. corporetione. 
amaM O Ml'w m ii. end farmers. 

22. Assooefron CaawmMrrraa^Corwminiiiaa where momoara belong on tha beats of affiliation rubor than employment, such aa unions and 
profaastona) societies, 

2X Qo mmm mt Commumtvt— Cornmunittee designed to adminlstar government ragulst ions and sarWcaa. lucft aa city hall, itatt dapanmant 
of education, and tegisisJlvacornmunMios. 

24 rvqnpovammarifaf /FuW#c Serwct Commumtm Other man fha MsfffWion Producing the Outcome— Nonprof ii service organization* . such 
aa schools, hospitals, weHemegenctee. philanthropic foundations, colleges (othar than me college producing tna outcome), and rtsaarch 
orgrnuaiiona. 

25. tottrtuhonot uwitutm** untt Frodudng ma Ouicoma— Tha poataaeondary aducaiion insiHution and/ or units within that institution that 
ana per cew ed aa ma producers facHttator of tha outcome* s) of concam 

26. Omar fmeraef-fleaed Communinoo- An asamof* would ba an ad hoc coalition task f orca of mpreeentattvne from two or mora of tha ebovu 
erooo. 

30. 0eoeva***«ased Commumfree-TWe eatagory rafara to largo groups daf mad on tha basis of function* ram tort* boundorm. 

31 Loc* Community— A township. city, county , metropolitan ara*. or othar typa of locality having particular boundartas If is not nacaasariiy 
raatrictad to tha tagat or jurisdictional boundary, but tna functional ona in which tha impact of tha institution is (or ihouid ba) diract ly and 
physically raft. Tha boundariaa «M vary with tha institution/program and ou'eoma of concern 

32. Th+$t*9> 

33. A AsofO*— An aggragat ion of stataa or parts of stataa. ^ 
34 Th*N*ton 

30. An tntomohonot Community 

30. Othot Oaagrapmc teted Communitm—An eiempie would ba a rasaar c h diacovary that affacts primarily paopia living tn iha coidaat 
latttudoe, or whara It snows hearth/. 

40. Aogragaras of rssjjj Ttin calagory rafara to suopopulaifons of paopfa dittmgwtfiod fry perrrcu/er cherecrerfsfics fhar may mdfeate common 
concerns, /weds, or warns, but who do not nacaasariiy hava a common intarast or mission, and inarafoi a do not const It ut a communities 

41. Aavary Law Sufaacau fafi nn Suoooputatloni daflnad according to level of ability/ prof iciancy on ganarai intellectual functioning or 
apaaific sanis— for aiampia, gifted, typical, disadvantaged, or smiled, semi-urned. unearned. 



42. 


Ago$uopo0uiohont 
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44. 


income Leva* Su*ooputot*on% 
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Occupenon Suoooovittont 
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Fnyvc* OHobHtty Condition Sudoomumtfont 
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So* Subpoputottont 


at. 


Other Sue* A p props fee 



90. 0»m Auiimct—tsmw** *ouM bo in* naiurd tnvMonnwni inn it ifroctod by uni«o«iity-seonMrod roiuren (which in turn would o* 
tipaciad to haw Knoaeii on mditnen men n tndtvtowali v~ communitioil tnd oooulotioni of *nim«i ;iucn u tno tmmaii iltoctod by 
Wlortt to hood dNMitd totem l<om Boeomtng oiiinct or By tho a»»*opmwH of vowmtry modwinn) 



a Repr1nted from terming, Lee, M1cek, and Service (1977, p. 24), where the focus 
was on "audiences"— Individuals, groups, communities, organizations, etc. receiving 
or being affected by particular outcomes of concern. This taxonomy constitutes the 
"audience dimension" of the NCHEMS Outcomes Structure. d£§j QQpy AVAILABLE 



Kaufman, Corrigan, and Johnson (1969), the focus in the needs assessment 
literature has tended to be entirely on discrepancies in outcomes or results. 

More recently, some discussion has centered on whether a discrepancy 
concept of need is adequate or necessary for an effective and productive 
needs study. Scriven (1977), for example, refers to such a definition as 
the primary reason that the needs assessment models employed in the past by 
evaluators "are farcical and decisions based on them are built on soluable 
sand [p. 25]." One problem with the discrepancy concept of need is that it 
has led many needs assessors to equate wants or demands with needs. Wants 
may very well be indicators of the presence of need, and especially if they 
turn to demands or expressions of anguish* but there still may not be needs 
present. For example, people may want or demand something merely because others 
have it, because it will attract attention, or to keep others from getting it. 
In other words, the real need may not be the expressed need. Conversely, people 
may have needs and not realize them, or they may recognize the need but not be 
willing to act upon what the need implies. For example, a student may recognize 
the need for counseling but not want to request it, or be unwilling to take 
the necessary steps to follow through on the counselors suggestions and insight. 

It is important to identify and analyze wants and demands in an assess- 
ment of need. However, most program needs assessments have not gone beyond 
that; they have equated wants and demands with needs, fteeds assessors should use 
any additional evidence that they can find, or develop, to help them determine 
whether the wants and demands indicate "real needs" and whether other needs are 
present. 

Another problem with the discrepancy or deficiency definition occurs when 
the need has been fulfilled. In discrepancy-based needs assessment r "met needs" 
are not considered to be needs, even if the deficiency would reoccur were the 
support withdrawn that has helped the need to be fulfilled. Assessments of 



r o 
Ierlc 



126129 



need by student affairs personnel should identify and assess target group met 
needs as well as unmet needs, so that they are prepared to act should conditions 
change and deficiencies appear in those areas of current fulfillment. Or, to 
use the terms of Scriven (1971), they should focus on maintenance needs (need 
for "lots") as well as incremental needs (need for more). 

Scriven (1977) makes the additional point that just because someone would 
benefit from something does not mean it is a need. For example, a gift of one 
million dollars would benefit most people; but having less than a million dollars 
would not put most people into an "unsatisfactory condition." According to 
this view, a need is present only when an unsatisfactory condition exists or 
would exist if the need were not being met. But who decides what is an unsatis- 
factory condition, and what criteria should be used in making that judgment? 

A "need," as defined in this paper, is a combination of discrepancy and 
level of ecessity. The amount of both components should be judged by a 
relevant person or group using multiple objective criteria 'id methodologies 
that have been mutually agreed upon. The relevant person or group to determine 
and apply the criteria of when the necessity and discrepancy constitute ..sed, 
and the point at which needs are partially or fully met, depends on the situation 
and context. However, it is important to remember that the amount of need 
varies directly with both the level of necessity and the amount of discrepancy— 
the same amount of necessity with increased discrepancy means greater unmet and 
overall need, and vice versa. Furthermore, as Burton and Merrill (1977) have 
suggested, unmet needs can be met by lowering the threshold of necessity, 
closing up the discrepancy (overcoming the deficiency), or a combination of 
both. For example, a student affairs professional's need to publish a study 
can be reduced by lowering the expectations of the profession concerning 
publications, or by the person completing and publishing a study. 
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What Need? With What Types of 
Needs Should We Be Concerned? 

For any individual or group, there are many different needs that could be 
of concern. The next question concerns which types of need are most important 
to identify and assess for the person or group, and program. 

In their review of the literature on needs and needs assessment, Lenning, 
Cooper, and Passmore (submitted for publication) identified a number of 
categorizations of need. Many of these categorizations are pertinent to the 
concerns of student affairs members and can stimulate thinking regarding the 
process of making decisions about which needs to attempt to assess for different 
target groups. 

An early classification of needs was developed oy Murray et al . (1938) 
based on interviews with paid college student males at Harvard. Their work 
stimulated the development of various need for achievement or achievement 
motivation and need for affiliation scales. Some of these scales were objective 
and some projective in nature. Murray postulated twenty manifest (leading to 
overt action) and eight latent (not leading to overt action but to action 
imagination and fantasies) needs. 

Maslow's (1954) need hierarchy is another classification of needs that 
has greatly influenced student affairs professions. According to Maslow, 
needs at a particular level cannot be met until those lower in the hierarchy 
have been satisfied. There have been a number of potentially useful attempts 
to operational ize Maslow's formulation, such as the one by Groves, Kaholis, 
& Erickson (1975). 

Developmental tasks related to maturity and chronological age in which 
earlier tasks must be mastered or accomplished before individuals can move on 
to further tasks suggest a different set of needs. For example, Cronbach (1963) 
referred to several basic needs including: affection, adult approval, peer 
approval, independence, and competence and self respect. In an earlier and 
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more extensive formulation, Havighurst (1952) identified ten primary ordered 

developmental tasks for adolescents, eight for early adulthood, seven for 

middle age, and six for later maturity. 

It is also possible to examine needs in terms of classifications of goals. 

For example, Beatty (1976) has related needs to goal -state continua. Her 

prescriptive (or ascribed) needs for individuals are determined by societal 

norms and standards, while her motivational needs are determined by the individual's 

goals. Lenning (1977a) has compiled numerous classifications of educational 

goals found in the literature for individuals, society, and individuals plus 

society. In some the categories are very detailed and narrow in focus, while 

in others the categories are broad. 

Related to goals are outcomes. As mentioned earlier, needs assessment 

studies have generally focused on needs in terms of outcomes or results. For 

example, Chickering (1969) identified seven developmental vectors for college 

students, each of which had two or more subcategories of outcomes: autonomy, 

competence, identity, integrity, interpersonal relationships, managing emotions, 

and purpose. Bowen's (1977) identification of numerous types of long-term 

impacts on graduates and society is another example. Figure 2 shows a generic, 

neutral, and comprehensive taxonomy of types of possible outcomes that is is 

believed could apply to any of the target group categories of Figure 1. Many of 

those types of outcomes could potentially constitute important student affairs 

program needs . 



Figure 2 goes about here 

Environmental needs and the processes needed to bring about the desired out- 
comes in this area must also be considered. For example, Chickering (1969, 1974), 
Aulepp and Delworth (1976), and Baird (1976) have discussed educational environ- 
mental factors and processes that can best bring about particular types of 
student outcomes. In addition, Lenning et al (1974a, 1974b) compiled a compre- 
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Figure 2 

CATEGORIES AND SUBCATEGORIES OF OUTCOMES IN THE 
TYPE-OF-OUTCOME DIMENSION OF THE NCHEMS OUTCOMES STRUCTURE 3 



Co^ ! Ngmo«f 6n!,!y Bain ° Ma,nf *' nt<J or Changed 


CodoNumbcr E » tl! y Be,n « Maintained or Changoo 


1000 ECONOMIC OUTCOMES 

1100 Economic Access and Independencs Outcomes 
1110 Economic Access 

1 120 Economic Flexibility. AdeptaoiMy, end Security 
1 130 Income end Standard of Uvmg 

1 200 Economic Resources and Cot t u 

1210 Economic Costs and Efficiency 

1220 Economic Resources (including employees) 

1300 Economic Production 

1310 Economic Productivity and Production 
1320 Economic Services Provided 

1400 Other Economic Outcomes 


2000 HUMAN CHARACTERISTIC OUTCOMES (continued) 
2760 Power and/or Authority 

^77rt uk ^rSnni or 1 i fn ^ii^^ri* 1 * 
Ci f U JOD. wtnOQI, Oi wlIC QUWbtlaS 

2780 Otner Status. Recognition, and Certification Outcomes 

2800 Social Activities end Poles 

2810 Adjustment to Retirement 
2820 Affiliations 

2830 Avocstionai and Social Activities and Roles 
2840 Career and Vocational Activities and Roles 
2850 Citizenship Activities and Roles 
2800 Family Activities and Roles 
2870 Friendships and Relationships 
2C80 Other Activity and Role Outcomes 

2900 Other Human Characteristic Outcomes 


2000 HUMAN CHARACTERISTIC OUTCOMES 

2100 Aspirations 

2110 Desires. Aims, end Goals 
21 20 Oiaiikes . Likes, end interests 
2130 Motivation or Orivo Level 
2140 Other Aspirational Outcomes 

2200 Comoetence and Skills 
2210 Academic Skills 

2220 Citizenship and Family Membership Skills 
2230 Creativity Skills 
2240 Expression and Communication Skills 
2290 intellectual Skills 

2260 Interperscral. Leaoership. and Organizational Skills* 
2270 Occupational and Emp.cyabihty Skills 
2230 Physical and Motor Skills 
2230 Other Skill Outcomes 

2300 Morale. Satisfaction, ana Affective Characteristics 
2310 Attitudes srd Values 
2320 Beliefs. Commitments, and Philosophy of Life 
2330 Feelings and Emotions 
2340 Mores. Customs, ano Stands/as of Conduct 
23d0 Other Affective Outcomes 

2400 Perceptual Characteristics 

2410 Perceptual Awareness and Sensitivity 
2420 Perception of Self 
2430 Perception of Others 
2440 Perception of Things 
2450 Oth*r Perceptual Outcomes 

2500 Personality »r*d Personal Coping Characteristics 
2510 Advent urousntss and Initiative 
2520 Autonomy and Independence 
2530 Dependability and Responsibility 
2540 Dogmatic /Open-Minded. Authontanan/Oerr.ocratic 
2550 Flexibility and Adaptability 
2560 Hsbita 

2570 Psychological Functioning 
2560 Tolerance and Persistence 
2500 Otner Personality and Personal Coping Outcomes 

2600 Physical and Physiological Characteristics 
2610 Physical Fitness and Traits 
2620 Physiological Health . 
2630 Other Physical or Physiological Outcomes 

2700 Statue. Recognition, and Certification 
2710 Completion or Achievement Award 
2720 Credit Recognition 
2730 Image, Reoutation. or Status 
2740 Licensing and Cent f icatton 
2750 Obtaining a Job or Admission to a Follow-up Program 


3000 KNOWLEDGE. TECHNOLOGY, AND APT FORM OUTCOMES 

3100 General Knowledge and Understanding 

3110 Knowledge and Understanding of General Facts and 
Terminology 

3120 Knowledge and Unoers tending of General Processes 
31 30 Knowledge and Understanding of General Theory 
3140 OtnerGenerai Knowledgeand Understanding 
3200 Specialized Knowledge and Understanding 

3210 Knowledge And Understanding of Specialized Facts 

and Terminology 
3220 Knowledge and Understanding of Specialized 

Processes 

3230 Knowledge and Understanding of Specialized Theory 
3240 Other Specialized Knowledge and Understanomg 

3300 Research and Scholarship 

3310 Resesrch and Schoiarshio Knowledge and 

Understanding 
3320 Research and Scholarship Products 

3400 Art Forms and Works 
3410 Architecture 
3420 Dance 

3430 Debate and Oratory 
3440 Oramfi 

3450 Literature and Writing 
3460 Music 

3470 Painting. Drawing, and Photography 
3480 Sculpture 
3490 Other Fine Arts 

3500 Other Knowledge, Technology, and Art Form Outcomes 


4000 RESOURCE AND SERVICE PROVISION OUTCOMES 

4100 Provision of Facilities and Events 
41 10 Provision of Facilities 
4120 Provision or Sponsorship of Events 

4200 Provision of Direct Services 
4210 Teaching 

4220 Advisory and Analytic Assistance 
4230 Treatment, Care, snd Referral Services 
4240 Provision of Otner Services 

4300 Other Resource and Service Provision Outcomes 


5000 OTHER MAINTENANCE AND CHANGE OUTCOMES 
5100 Aesthetic-Cultural Activities. Traditions, and Conditions 
5200 Organizational Format. Activity, and Operation 
5300 Other Maintenance and Change 



Reprinted from Lenning, Lee, Micek, and Service (1977), page 27. The fourth-level 
Categories, into which any of the categories listed here can be divided, are 
"maintenance" (a fourth digit of "1") and "change" (a fourth digit of "2"). 
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hensive review of nonintellective factors found in the literature to correlate 
with various types of educational outcomes, such as student grades, persistence, 
motivation, attitudes and values, social ski i 1 s and participation, and confidence 
and self concept. 

Needs can also be related to problems. For example, each of the 11 problem 
areas for which the college student form of the Mooney Problem Check List (Mooney 
and Gordon, 1950) has scales can imply particular needs: (1) health and physical 
development, (2) finances-living condition-employment, (3) social and recreational 
activities, (4) social -physical relations, (5) personal -psychological 
relations, (6) courtship-sex-marriage, (7) home and family, (8) morals and 
religion, (9) adjustment to school work, (10) the future-vocational and 
educational, and (11) curriculum and teaching procedures. The same is true 
of the "adult" form of the check list, which has the following nine scales: 
health, economic security, self-improvement, personality, home and family, 
courtship, sex, religion, and occupation. Related to the above, but coming 
at it from the opposite direction, Campbell and Markle (1967) developed 
techniques for transforming educational needs into well-defined problems. They 
contend that educators need carefully designed problem- formulation training 
in order to translate needs to problems effectively. 

Bradshaw (1972) and Burton and Merrill (1977) have proposed an additional 
typology of needs that could be useful in student affairs programming. The 
categories in this typology are: (1) normative needs, (2) felt needs, (3) 
expressed needs, (4) comparative needs, and (5) anticipated or projected needs. 

Still another potentially useful classification of neec* ft*r student services 
consists of basic (primary, root, or underlying) human needs versus secondary 
(derived, learned, or deduced) needs. Monette (1977) has labeled these two 
categories "in-.ate needs" and "acquired needs." 
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What Approaches and Procedures Should Be Used 
for Conducting Student Affairs Needs Assessments? 
Conducting needs assessment is an important activity related to student 
affairs program evaluation. Few suggestions have been put forward as to how 
such assessments can systematically and effectively be carried out within 
student affairs, however, expecially for assessing target group needs. This 
section will discuss how to conduct such assessments, with the focus first 
being on the assessment of individual needs and then on the assessment of group 
needs . 

Assessing Individual Needs 

Since tf~i student development profession emerged out of a recognition 
of the students 1 needs outside of the area of intellectual achievement, any 
program designed to identify the needs of individual students more effectively 
is clearly of central importance. Assessment procedures utilized in this 
context can assist a student in identifying current needs and clarifying 
strategies for future change. Data relating to individual personality 
characteristics * perceptions, values, goals, and interests can provide a 
framework upon which a student can move towards continual growth and development. 

One such needs clarification model was initiated at Azusa Pacific 
College as a means of enhancing planned growth among individual students. The 
strategy was formulated as part of a research project conducted by graduate 
students in the M.A. in Social Science with an Emphasis in Student Development 
program during Spring 1977. It was initiated on a pilot project basis during 
the 1977-78 academic year. Graduate students in Student Development experimented, 
experienced, and revised the procedures based on a two-semester, three credit 
course. 

The major components of the project will be described in this section as an 
example of a needs clarification procedure. In the first step, a conceptual 
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I framework for the project was created oy examining various developmental and 
n learning theories. Chickerlng's (1969) seven vectors of development for 

■ college students were instrumental 1n identifying the six-stage model that was 
I adopted. The areas chosen for examination of personal needs among college 

students Included: Personal /Emotional , Social /Interpersonal , Intellectual/ 
I Cognitive, Values/Rel 1g1ous Orientation, Career/Life Planning, and Physical 
m Fitness. 

The next step involved the selection of appropriate assessment tools 
I for aach of the six areas. Criteria for selecting the best available 
assessment tools were established as follows: 

■ 1. Assessment tools should measure each area of need in terms of 
m a clearly defined conceptualization of that need. The tool 

must lend Itself to evaluating the area of need perceived. 
I 2. A comparative analysis should be made between the alternative 

m assessment tools taking into consideration these factors: 

■ - Degree of congruence between stated purpose of Instrument 
B and the needs assessroent/clarifi cation purpose. 

- Availability of appropriate instrument. 
| - Research data regarding the Instrument's validity, reliability, 
m and standardization. 

■ - Data concerning successful usage by others for similar purposes 
I based on recomr nidations, interviews and research. 

- Costs Involved— Funds available. 
§ For the experimental model purposes , particular assessment Instruments 
g were chosen for each of the six areas of student development as follows: 

■ Personal /Emotional (California Psychological Inventory), Social/Interpersoa 1 

■ (Fundamental Interpersonal Relations Inventory), Values/Religious Orientation 
("The University of Wisconsin Survey Instrument and University and Society: 
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_ Student Perspectives" and selected values clarification exercises), Yourself 
Halfway and Values Clarification) . Cognitive/Intellectual (Omnibus Personality 

I Inventory and Learning Styles Questionnaires), Career/Life Planning (Hall 

Occupational Orientation Inventory), and Physical Fi>*ess (3 Day Food Intake 

I Analysis, 1.5 Mile Running Test, 1 Minute Sit-up Test, Trunk Flexion and 

m Extension Tests, Skin Fold Test). 

In order for the student to gain significant insight in*o his or her own 

■ development, and to consequently be able to establish personal goals, it is 
necessary that the assessment process include more than collection of data. 

I Data must be interpreted to the student and guidelines provided to help the. 

m student '"tilize insists gained from the personal needs and developmental 

goals clarification process. 
I After examining other needs assessment programs with similar goals, a 

five-part sequential plan was formulated to facilitate the process of clarifying 

■ student needs. It is important to note that a variety of procedures can be 
ft utilized as background preparation for each of the assessment areas. Pertinent 

bibliographic materials should b~ included that will provide students with 
I background information as well as material for future references. 

- 1 . Data ColTection 

■ The first stage in needs assessment should include an explanation 
I of the need(s) being measured, the instrument(s) to oe utilized and 

the objective(s) that are to be realized after the progress is 
I complete for each individual area. 

I 2. Scoring 

The second stage consists of obtaining test results. Scoring 
I methods include: machine, professional psycomotrist or counselor, 

or calf scoring. 
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3. Interpretation 
The third stage is the most crucial aspect of assessing individual- 
student needs. It involves communicating the test results to the 
student so that self -understanding is Increased* Interpretation 
techniques vary according to the nature of the instrument used and 
the need being assessed. Techniques may Involve the student using 
a test manual as a guide to Interpret test scores or an Individual 
interview or small group discussion with a counselor, faculty 
member, or paraprofessional who 1s competent 1n testing. 

4. Goal Setting 

The fourth stage of individual -student needs assessment in actuality 
is a significant factor in determining the success of the previous 
stages. Assuming that the student has gained increased self -understanding 
through the previous stages, this stage should assist the student 
in adapting Insights and formulating developmental goals based on 
that understanding. The degree to which this is accompli shad 
determines in large part the success of the assessment program. 
Again, the means used to accomplish this goal setting stage will 
vary according to the needs assessed and instrument(s) used. One 
helpful device may be the use of an individualized worksheet 
or workbook. (Deddo and Thuveson, 1977) 

5. Evaluation 

The final stage of the needs assessment program involves evaluation 

of the process from the scent's perspective. This enables 
the student to reflect and react regarding the aspects of each 

area assessed. In terns of the goals stated, to what degree did 

the process actually imp* t insight to the student concerning his/her 

own development? In addition to providing a sound basis for future 

development, this information should be utilized by staff for 

revision of the assessment program. 
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It 1s recommended that some sort of evaluation questionnaire be 
administered to each student in order to facilitate feedback in 
pertinent areas of concern. (McAleenan and Deddo, submitted for 
publication) . 

A schematic representation of this procedural model for student needs 
clarification is shown in Figure 3. 



Figure 3 goes about here 



The length of time, availability of staff, resources and cost are all 
factors that need to be taken into consideration in adopting such a program. 
Ba^ed on student evaluation of the pilot project at Azusa Pacific College, it 
is -important to note that the interpretation phase is often complex and time 
consuming. Test data results often vary from a student's previously held 
assumptions and beliefs. This experience can be highly threatening and often 
requires time for analysis and synthe: .s through the help of a peer group and/or 
staff member. The amount of discrepancy and level of necessity for resolving 
the need are complex variables that must be carefully digested in order to be 
acted upon appropriately. The mere acquisition of new data is not enough to 
promote student growth on an Individual basis. 

Students must be guided through a process of integratin new 
information with previously held values. Students affairs professionals 
and paraprofessionals have a clear responsibility to follow a student 
through all the stages of an assessment process. This may also necessitate 
establishing workshops on goal setting techniques since students rarely 
have had experience in the process of setting personal goals. 

Assessing the needs of individual student affairs staff members is also a 
dimension of evaluation that must be examined. As with student assessment, a 
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A PROCEDURAL MODEL FOR STUDENT NEED CLARIFICATION 3 
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system for interpretation, reflection, and discussion regarding data must 
be included. A strategy such as the one outlined for students can also be 
utilized effectively with staff, on an individual or^ group basis . Another 
example of a viable system for assessing staff member needs is outlined in 
a monograph by Kiersy and Bates (19 ) entitled Results Systems Management . 
It is an effective procedure to assist staff in identifying priorities and 
individual goals for personal and professional growth during a given month, 
semester, or school year. 

Assessing Target Group Needs 

Although some of the considerations and activities are the same or 
similar, assessing group (program) needs involves different processes than 
assessing the needs of an individual. First we will diicuss how group needs 
information can be applied to student affairs program evaluation, planning, 
and policy formulation- Next, various overall approaches and orientations 
to assessing group needs will be surveyed. 

Group Needs Assessment Application . A primary purpose for assessing 
group needs in student affairs is to provide input and guidance for program 
planning, management* and evaluation decisions, including policy formulation. 
Figure 4 presents a graphical overview of a general framework for program 
planning, management, and evaluation that illustrates how needs assessment 
fits into the total enterprise. In this framework, constituent (e.g., students 
and student affairs staff) needs are identified and assessed. Needs assessment 



Figure 4 goes about here 

results are used to aid in the process of acquiring resources for the program 
(funding agencies and financial management people must be convinced the 
resources are needed). Furthermore, the needs and need priorities determined 
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Figure 4 

ONE POSSIBLE FRAMEWORK FOR PROGRAM PLANNING, MANAGEMENT, AND EVALUATION 
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m in the needs assessment, are examined in light of the mission of the institution 

■ and of the program, and broad outcome and process goals are generated. (Unmet 
needs will suggest improvement types of goals while met needs will suggest 

■ maintenance types of goals). The goals are now transformed into more specific 
m goals. Next the needs and their context (e.g., their root causes and analyzed 
* amenability to solution) will provide input for evaluating the goals, refining 

■ them, arriving at goal priorities, and transforming each goal to specific and 
concrete desired objectives that can directly guide planning. The objectives, 

I 1n the light of what we know about the needs, their priorities, and their 

contexts, are then applied to evaluating alternative courses of action, 

1 activities, and processes. Best courses of action and activities to meet 

■ the needs/goals/objectives are then formulated, interaction with resource 
providers and allocators takes place, resources are allocated and reallocated, and 

I final objectives are planned. Then, the process objectives are carried out in 
^ ways that are continually ameliorated by what is known about the needs and 
™ outcome goal' determined originally, and evaluative date are collected, 
t Finally, evaluation activities are conducted to determine whether the needs 

and goals are being met, how effectively they are being met, and what improve- 
I ments are needed. 

■ This discussion has focused separately on each step of the planning, 
management, and evaluation process outlined in Figure 4. As indicated by 

I the dotted lines extending back to each preceding phase from the evaluation 

phase, in that diagram, it is a cyclical process that repeats itself 
I periodically, with the evaluation phase providing helpful feedback to the 

■ next iterations of the earlier phases. Knowledge about program constituent 

needs is thus related directly to earlier evaluation results, and interpreted 
| accordingly. 
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■ Group Needs Assessment Orientation and Approaches . A number of group needs 
assessment models or approaches have been developed for use in the educational 

I setting, many which have practical implications for conducting assessments of 

student affairs program needs. Some are primarily subjective in their 
I measurement and analytic approach while others are primarily objective. Some 

■ use only data collected for the study, while others make use of secondary data 
analysis. Although most models developed up until now focus entirely on 

I discrepancy need, this is not true of all of them (e.g., Coffing & Hutchinson's 

model [1974]). 

1 Some approaches are interactive in nature (the process includes much 

■ Interactive and direct involvement by the people whose needs are being assessed) 
while others only involve constituent groups in providing self-reports of 

I needs. Similarly, there are approaches that rely only on self-report and 

^ those that rely mainly on other data. In addition, some approaches use 

• multiple measures, sources of data, and data collection methods, as constrasted 
t to models relying on unitary measures, sources of data, and collection methods. 

Some needs assessment approaches are goal-driven (needs are searched 
I out in terms of what is Intended for the program ), while others have "goal-free" 

I aspects that allow other needs to also come to the fore which allows needs 

data to assist in evaluating, developing, and refining goals. Interpre- 
I tation in the first type of approach is deductive in nature; it is inductive 

1n the second type of approach. Concerning the goal-driven approaches, some 
I relate needs to the goals of the constituents or clients while others relate 
| needs to the goals of the educators or student affairs staff members implementing 

the program. Some models include goal setting as a part of the needs assess- 
I ment process while others (such as the framework of Figure 4) view goal- 

setting as a separate process to which the needs assessment effort provides 
I Input. 
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Some group needs assessment approaches have a diffused breadth of focus 
and Identify general needs ("wide-band" study), while many needs assessment 
approaches 1n postsecondary education are specialized and specific in their 
focus and identify detailed needs ("narrow-band" study). Some approaches 
only identify needs, others also determine need priorities, and a few attempt 
to provide meaningful understanding about the needs and why they exist. 
Some approaches limit cheir attention to identifying and assessing current 
needs, while others concentrate on projecting future needs. 

Group needs assessment approaches can also be differentiated according 
to the way that they are applied to determining program priorities and to 
decision-making. Some apply one-dimensional decision roles, while others use 
multiple and concurrent thresholds. Some use simple and rough decision techniques 
while others make use of sophisticated optimization procedures. Some integrate 
the various data while others consider each type of data separately. 

A word should also be said about three types of approaches differentiated 
by Scriven (1977). The first two are discrepancy models—the first focusing 
on discrepancies in personal perceptions of "what is" and "what should be," 
and the second focusing on the discrepancy between "what is" for this group 
and the average or norm for some aggregate such as across the state or nation. 
Scriven castigates both of these approaches. Instead he favors the medical 
model, which examines changes in conditions when deprivation occurs. He 
also makes a useful distinction between what he calls "performance deficits" 
and "treatewnt deficits" (treatment needs implied by the existence of 
performance deficits). Scriven cautions that special care must be taken 
when deciding whether there is a treatment deficit and what the deficit 
involves. For example, a finding that fifty percent of the students 
entering a college are dropping out during the first term (performance 




deficit) can imply a large number of potential treatment deficits (inadequate 
information is being provided to prospective students, admissions counseling 
is lacking, orientation program inadequacies, insufficient publicity available 

for student services resulting in nonusage, student "program services and 

staff Inadequacies, instructional -related dissatisfactions, etc.). 

Conducting Group Needs Assessment . The conceptual discussion earlier, 

plus what was reported in the preceding subsections of this section, suggest a 

number of "good practice" procedures for assessing needs. Furthermore, various 

procedural handbooks and manuals that give step-by-step procedures for conducting 

group needs assessments have been developed. Those that seem most pertinent 
to group needs assessment in student affairs (some focusing on enrolled student 

needs and some on nonstudent needs) Include: American College Testing Program 
(1976), Aulepp and Delworth (1976), Hays and Linn (1977), Higher Education Manage- 
ment Institute (1977), Lenning and Cooper (1978), and Mager and Pipe (1970). A 
review of English and Kaufman (1975), Coffing and Hutchinson (1974), and Witkin 

(1975) could also prove helpful. 

1. The first step in planning for and conducting an assessment of 
student affairs program needs (group needs) is to decide on and 
outline the specific purposes, objectives, uses and users to be 
served by the assessment. Among other things, this should involve 
soliciting input on this topic from representatives of a wide 
array of constituents and potential users of the information to be 
generated by the study. 

2. The second step is to determine exactly which populations and groups 
should be of concern for the specific program in question. Groups 
and subgroups should be carefully defined. Start first with those 
student subgroups currently being served, according to program 
records. Then try to think of other groups, students and otherwise 
that should be served by the program, but haven't been served. 
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■ 3. Third, decide which particular groups identified in 2 are feasible 

to study at this point in time and which groups should be studied in 
| a later year. (Costs, funds available, and staff resources and time 

_ should be considered carefully in making these decisions.) For each 

" group and subgroup selected for focus, try to develop a preliminary 

I understanding of each, and tr~ needs of its members, through prelimi- 

nary open-ended interviews within each group, and with pertinent others 
I (such as instructors in the case of students). 

■ 4. Based on the stated purposes and mission of the program, and infor- 
B mation gathered in Steps 1-3, decide on the types of needs that 

■ should especially be of concern for each group, in the study. 
5. Considering usefulness, validity, and reliability of the data, 

1 select multiple measures or indicators and data collection sources and 

m procedures appropriate for each type of need. (The instruments 

* used do not need to be as reliable as those for use in assessing 

■ individual needs to still be useful; whereas, a test instrument 

for individual use should have reliabilities above 0.8, they can 

I be as low as 0.6 for group use and have adequate reliability.) 

m Some of the data from instruments will be "hard" (objective, 

■ accurate, and reliable measures) while other useful data will be 

I ' aOft" (subjective, perceptual, nor fully reliable or clear cut in what is 

being measured). Multiple measures and data collection methods 
| are desirable, whenever feasible, because where one is weak 

■ another may be strong, and vice versa. In addition to psychometric 
tests, survey questionnaires, or interviews, additional methods 

I should be considered for gathering data. Lenning (1978) found almost 

50 such additional methods mentioned in the literature that have 
I been found by various professionals to be useful and feasible for 

| particular uses and contexts; and most of them ^ould be applicable 
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in specific situations to assessing student affairs needs. "Secondary 
data"— data such as in institutional or community records that are already 
available (collected for other purposes)— should also not be overlooked. 
\ group of them examined simultaneously could yield effective 
u.iobstrusive (non-direct) measures that suggest the presence or 
absence of particular needs. 

Test out the instruments, procedures, and methods, using a few 
members of each group, and modify as indicated. Next collect the 
planned data. 

Analyze the data, in common-sense, logical ways that will suggest 
what needs are present. This includes the computation of simple 
statistics such as frequencies, percentages, means, and standard 
deviations; and the use of simple statistical tests such as t-tests 
eid chi-squares. (Frequency patterns are as important as averages, 
becijse change may be up for some individuals at the same time it is down 
for other members of the group.) It also involves integrating different 
types of data, including hard and soft data, and conducting profile 
analyses for examination of similarities, discrepancies, and patterns 
(see the model for assessing guidance needs developed by the American 
College Testing Program [1976] for an effective graphical way of doing 
this). Sophisticated analyses such as multiple repression analysis, 
analysis of variance, analysis of co-variance, and discriminant an^ysis 
can be very helpful, but will often not be necessary. The needs identi- 
fied will need to be rank ordered through applying multiple criteria 
such as amount of necessity, how vital >s the area of need for effect! e 
functioning, the amount of discrepancy, etc. In addition, through pro- 
file analysis and logic, hynoth.- s that can help in providing an 
understanding of the probable reasons for the occurrence of the various 
needs should be formulated. 
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8. Interpret the results and communicate them to those identified in 
the beginning as users of the needs assessment results. Separate 
hHef, concise reports tailored to each pertinent decision maker's 
(or type of decision maker) needs that speak to specific problems 
of concern to him/her should be prepared and distributed to those 
persons. 

Conclusion 

Although most needs assessment models have defined needs as discrepancies 
betweer "what should be" and "what currently exists," it is more helpful 
to think of needs in terms of a combination of discrepancy and necessity 
where there are b~*h met and unmet needs. The amour t of need should be judged 
by a relevant person or group using multiple objective criteria and methodology 
that have been mutually agreed on. 

Needs assessment of individuals is designed to raise awareness and provide 
guidance for a particular person concerning his or her own development or advance- 
ment, or for individuals who are working with or providing guidance to that 
individual. Needs assessment of groups, on the other hand, is designed to 
guide the development and improvement of programs—including planning, 
management, and evaluation decisions, and policy formulation. Group need data 
in conjunction with program mission information are also an aid in the process 
of acquiring resources for the program* Although they are similar in overall 
concept, the strategy and techniques for conducting nf ^s assessment of 
individuals and of groups differ operationally. 

The purpose of evaluation is to determine whether or not defined goals are 
being met and how they can be better met. If the goals are not being met 
adequately, the goals themselves or the ways of meeting them may be redefined 
as necessary, based on the findings. But underlying it all should be needs 
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assessment. Adequate needs assessment procedures can ensure that the 
evaluation process begins on the proper basis. To reiterate, it is important 
to note that needs assessment is an important component of the total evaluation 
process. The use of needs assessment techniques can enhance the value and 
benefits of that process. Program evaluation results are not very useful unless 
they can be related to needs. 
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ASSESSMENT AND EVALUATION 
RELATED TO STUDENT SERVICES 

Oscar T. Lenning 



Traditionally a chapter on assessment and evaluation would occur last in 
a section of a book such as this because such activities would be considered after- 
the-fact in relation to the activities and processes discussed in the chapters 
that follow. Assessment and evaluation cannot be very effective or have a 
major impact on improving those other processes, however, if they are after-the- 
fact activities. In fact, evaluation planning should occur along with the very 
earliest planning for an operational program, whether it is for a student 
services program or some other kind of progran. Therefore, because such activities 
can apply to all of the types of programs discussed in the following chapters 
and should be considered in an early stage of planning for each of them, it is 
appropriate that this chapter on assessment and evaluation occur first. 

It has increasingly been recognized that planning for evaluation must be an 
integral part of planning for a process and program, and should take place before 
the process begins in order to insure that all of the needed data will be collected 
at the appropriate time prior to and during the operational phase. This is 
true of end-of-period or enH-cf-tryout summative evaluation , the traditionally 
accepted form of evaluation that is used to suggest whether the program should 
be continued, terminated, or revised. An example of one of the problems that 
early planning prevents, lack of adequate baseline data, is discussed by Anderson, 
Ball, Murphy and Associates (1976) as follows: 

Evaluations are frequently undertaken without adequate 
baseline data. Sometimes, through lack of foresight, an 
evaluation is not envisaged until after the program is *n 

^Special appreciation is hereby expres\ed to Philip Beal for critiquing this 
q paper and providing helpful suggestions. 
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operation. In other instances, a program is considered 
so urgent that it is implemented before an evaluation can 
be planned and baseline data obtained; some of the War 
on Poverty programs begun in the 1960's fall in this cate- 
gory. For whatever reason of neglect or priority, when 
baseline data are unavailable, evaluation results are 
likely to be equivocal and to stimulate considerable 
argument and confusion about whether the program was 
effective, (pp. 42-43) 
A potentially more useful corcept of evaluation than summative is called 
formative evaluation (Scriven, 1967). Formative evaluation occurs during the 
development and operation of the program— and continually suggests how the program 
should be modified in order to improve it— so planning for it at an early stage 
of program development is especially crucial. 

One component of evaluation (no matter what type of evaluation) is always 
assessment, as will be discussed shortly. As such, assessment leads to, and 
precedes, the activities and processes described in the later sections of this 
chapter. On the other hand, some assessment is not a component of evaluation— 
for example, assessment to guide strategy and activities when counseling with 
individuals. 

Assessment Versus Evaluation 

In some cases evaluators archer their own data on which judgements are 
to be based, while in other cases tney utilize data gathered and analyzed by 
others in measurement and asse r ,,ient phases. Thus, the roles and activities 
are not the same for assessment as for evaluation, nor are the skills and 
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competencies needed to perform those roles and activities (although there is much 
overlap). This section discusses the differences between assessment and evaluation. 

Assessment is a term made popular by Henry Murray and his associates 
in the late 1930' s , who referred to it as the appraisal of individuals. 
As such, we can assess a person's physical and emotional characteris- 
tics, personality, needs, behavior, competencies, development, 
performance, educational and occupational progress, readiness for further 
education, quality of life, etc. This can be done in general, or in terms 
of the person's role, for example, his or her role as a student or as 
a teacher. Because of Murray, counselors and other student services 
personnel have traditionally (and it is still true today) tended to associate 
assessment with individuals, and evaluation with groups and programs. 
Furthermore, they have tended to think strictly in terms of assessing 
students, whereas it is just as legitimate and pertinent to their 
profession to assess themselves— their needs, their skills, their attitudes 
and interests, their professional readiness, their performance, etc. 
Analogous to this is Menges* (1973) model for assessing the readiness of 
teachers for professional practice. 

Higher education authors outside of student affairs have often applied 
the term assessment to groups and programs— for example, Baird's (1974,1976) 
discussion of assessing educational environment, Doi's (1974) "assessing faculty 
effort," LaselTs (1974) conducting "assessment of internal decision events," 
and the Clark, Hartnett, and Baird (1976) assessment of the dimensions of 
quality in doctoral education." Furthermore, one often hears within education 
about assessment within even a larger ontext, such as a total educational 
institution, the total educational system in a state or for the nation as a whole, 
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and a community or society-at-large— for example, the assessment of 
enrollment patterning, educational environment, educational quality, 
educational impacts, and service to the community. And outside of education 
we hear about such things as assessing costs and ecological impact. 

Many people in education seem to equate assessment to evaluation, 
for example, Dressel (1976) and Lasell (1974). On the other hand, some 
equate assessment to measurement, for example, Harrocks (1964). And then 
there are those who equate evaluation to measurement, such as May (1975) when he 
defines guidance program evaluation as "the measurement of what is valued." 
Others take an intermediary position and view measurement as a component 
tal. of cessment, which is in turn a component of evaluation. This last view is the 
tal . relationship between measurement, assessment, and evaluation used throughout this 
pal . chapter . 

"Assess" is closely related to the term "assay," which means to examine, 
test, or analyze. Murphy (1976) has pointed out that "in its derivation, 
the word assess means 'to sit beside" cr 'to assist the judge' (p. 27)." 
From such a perspective, assessment refers to gathering the data, transforming 
the data so that it can be interpreted, applying analytical techniques, 
and analyzing the data in terms of alternative hypotheses and explanations. 
Based on such assessment, judgements about value, worth, and ways to improve 
can be made-the process of evaluation. Therefore, for purposes of this 
chapter, an assessment study includes measurement and analysis, while 
an evaluation study includes measurement, analysis, and judgement. One 
cannot have assessment or evaluation without measurement and analysis 
of some kind . 
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Another way to say the same thing is to use some words of Popham (1975). 

Ke defines measurement as "status determination, 11 and evaluation as "worth 

determination." Then, based on the above discussion, assessment is what 1 inks 
status determination to worth determination. 

Assessment Strategies and Considerations 
as They Relate to Student Services 

Lenning (1977a) has discussed eight uses of student assessment by college 
staff personnel. Three of them are definitely evaluation functions: grading, 
promotion, and merit awards; evaluating efficiency and effectiveness; and 
evaluating innovations. The others are less evaluative in nature: planning 
learning experiences; counseling and advising students; diagnosing student 
problems; appraising student readiness; and classifying and categorizing students. 
(Students can also usefully conduct their own assessments—of themselves, their 
needs, their environments, their activities, and their achievements.) Assessments 
of persons other than students (such as staff, graduates, the members of the 
community) can also be important in student affairs, and such assessments 
are potentially as diverse in their functions as are the functions of student 
assessment. 

In this section will be discussed general procedures for conducting 

assessments. Then the focus will shift to special considerations for different 
kinds of assessments, such as assessments of individuals versus groups, 

assessment by self versus others, and assessment of needs versus goals, versus 

achievements, etc. The discussion of both general procedures and special 

considerations will imply skills and competencies needed for conducting 

assessments. 

» 

General Assessment Procedures 

Payne discusses seven generally accepted stages or steps in assessing 
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cognitive and affective learning, and those stages appear to be applicable 
to all kinds of assessment: (1) specifying detailed goals and objectives, 
(2) designing the assessment system, (3) selecting measures and data gathering 
methods, (4) collecting data, (5) analyzing and summarizing data, (6) con- 
trasting data and objectives, and (7) feeding back results. Each of these 
tasks will be discussed from the perspective of a student affairs worker. 



■ Specifying Goals and Objectives . Student affairs program goals generally 
tend to be broad and vague abstractions that use such terms as "to promote 

I maximum development of the total self,' 1 "to promote self-actualization, 0 "and to 

develop realistic and independent decision making." If one is to have effec- 

■ tive guidelines for conducting actions to bring about accomplished ends for 

I students, a course, a proqram, or an institution, such qoals need to be 

transformed into concrete, observable, precise terms. For an example of one way 
I to do this, see Lenning (1977b, pp. 15-24). Although he was talking about 

_ university goals, what Conrad (1974) says about goals also can be stated in terms 

• of program goai^: (1) they are standards against which to judge program success, 

I (2) they provide a source of legitimacy for the activities of the program, (3) 

they define and order program needs, (4) they define the units of program outcomes, 
| (5) they identify the program's clientele, and (o) they define the relationship 

B between the program, the institution of which it is a part, and society. 

■ What we are trying to assess must relate to the goals and objectives of 
I the program, or individuals, being assessed. For programs, for example, 

the variables being assessed should relate to two different kinds of 
| program goals. First are outcome goals, the results or consequences 

_ that the program activities are intended to achieve. The other type of goals, 
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process goals, refer to how the outcome goals are intended to be achieved— 
the personnel, money, time, activities, techniques, methods, and tools that 
will be utilized and in which ways to achieve particular outcome goals. 

Transforming a goal into specific ends or objectives to be achieved is 
a difficult task for many people. The task of reaching maximum agreement among 
various staff members, anu constituents, concerning goals and goal priorities 
is also a difficult task, and may sometimes call for the application of 
special concensus rendering techniques, such as the Q-sort technique or 
the Delphi technique, if give-and-take discussion does not vield enough 
agreement. 

What has been said about the goals of the program (or of the individual in 
the case of assessing individuals) also applies to the goals of the assessment 
study. Those conducting the assessment must delineate clearly and in specific 
terms what is being assessed for whom, why the particular variables are being 
assessed, how the assessment data are expected to relate to individual or program 
goals, and how the assessment data are to be applied and used. 

Designing the Assessment System. Once the goals and objectives for the assessment 
effort are specified, work can begin on developing the assessment strategy and 
procedures to be used for accomplishing those goals and objectives. The success 
of the assessment effort rests on having an integrated, detailed, well-thought- 
out assessment design that effectively relates to the assessment goals and 
objectives. First the purposes of the assessment should be outlined, followed 
by a delineation of the context in which the assessment is to take place. The 
context includes factors within the program, institution, or other environments 
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that will either assist the assessment effort or constrain it, such as 
attitudes and values of staff or students, political pressures and situations, 
financial and staff resources available, time and space considerations, base- 
line deta that are already available, the diversity of the students using 
various student services, etc. Third to be outlined are the specific questions 
that need to be answered by the assessment to assist the particular problem 
solving, decision-making, or other purposes formulated for it. Next, decisions 
will be made and written down about: the information needed to provide evidence 
useful in answering the various questions, the available indicators tf-t can 
lead to such information, the relevant data sources that are feasible, whether 
sampling should be used and what kind, the data gathering and analysis strategies 
and procedures that will collect the proper data and convert it into pertinent 
information, the data interpretation strategy, and the feedback procedures 
for getting the information out to decision-makers and concerned others in an 
effective manner that promotes use. Finally, factors such as assessment costs 
and how the entire plan fits together into an integrated system will be 
considered and refinements or modifications made as necessary. 

The design must be realistic and f easible in terms of the costs and effort 
required, and must be effective in generating the information needed to answer 
the pertinent decision-makers 1 questions. It must identify and provide a 
rationale concerning which specific groups of students and other persons, and 
entities such as the environment, should be assessed. How the assessment strategy 
and procedures will vary by group or area should also be ascertained. For 
example, assessing older students 1 academic competencies using a standardized 
psychometric instrument designed for and normed on teenage college students 
would clearly be inappropriate unless it was tested and found to be valid for 
them also. 
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The design of an assessment for a student services program, or that is 

meant to provide assessment data for use by individuals, should be 

tailored specifically to the situation at hand. One may usefully be able to 

borrow ideas from another program or institution, but care should be taken to 

see how it should be customized to the local situation. The same is true of 

making use of general models, such as the General Integrative riodel developed 

by Pottmger and Klemp (1975, 1976) that involves the use of various tests and 

scales they developed for assessing the integration of life skills within 

students. Their model was designed for investigating how students process 

and integrate information, as contrasted to the storage and retrieval of 

information. Therefore, it can be used for assessing such things as how students 
thir* and react in various types of social situations, which is of major concern 

to most student affairs workers. 

Selecting Measures and Data Gathering Methods . In selecting measures and 
indicators, reliability and validity (with respect to the questions that need 
answers) are important criteria. So are factors such as ease of data collection, 
cost of data collection, ease of scoring and tabulating, appropriateness to 
the analytic procedures and tests that sre planned, etc. To illustrate the 
importance of such factors, this author recently heard about a new test battery 
that had been tested out with reported success, and that seemed to have adequate 
reliability plus good validity in measuring "real life" competencies. Since 
he was consulting for the evaluation of a nontraditional program emphasizin" 
the development of such competencies, he recommended that they try it out " " 
their evaluation. Fortunately, they tried it out in a pilot test basis because 
even though it admirably met the reliability and validity criteria, they found 
that it was extremely difficult for their people to administer and score. This 
case illustrates the importance of trying out measures and data gathering 
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■ procedures ahe=ui of time with small pilot samples of respondents similar 
to those in the study, before the final decision is made to use them in the 

■ full assessment study. 

Standardized paper and pencil instruments are often used in student and 

■ . program assessments; but although they may have good reliability and validity 

■ for what they purport to measure, they often do not measure what is specifi- 
cally of concern in the program. A good example of this is the Watson-Glazer 

I Critical Thinking Appraisal . It has excellent validity if one's concern is 

with the ability to reason critically in a passive manner, and it has good 

■ reliability. If one is concerned about the ability to apply that critical 

■ thinking proactively in "real life" situations, such as applying critical 
thinking to managing social confrontations, however, its validity is low 

I for measuring that competency. If one is considering the use of a standardized 

instrument, it is crucial to study the reviews in Boros (1972) as well as 
• any other available reviews (such as the ones in Measurement an^ valuation 

■ in Guidance , Educational and Pcychological Measurement , and NCME Measurement 
in Educatio n ) for assurance that it measures what needs to be measured. 

9 To get a paper and pencil instrument that measures specifically what is 

_ desired, one will often have to construct his/her own. There are an abundance 

■ of excellent texts on measurement theory, the development of norm-referenced 
S tests, and the development of questionnaires, with the variety being such 

that there are appropriate ones for experienced as well as inexperienced 
Q instrument developers. Whenever possible, locally-developed instruments 

m should build on similar ones developed elsewhere. Lang, Lehmann, and Mehrans 

■ (1967) have shown that revising items takes less time, effort, and expense than 
I developing them from scratch. Great care must be taken, ho /ever, to modify them 

appropriately for the new context. Another point which should be made here 
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is that locally-developed criterion-referenced instruments that focus on 
absolute level of performance or mastery should always be considered as an 
alternative to norm-referenced instruments (seeGronlund [1973] for help in 
developing such instruments). 

Most measures and indicators are more reliable and valid in some contexts 
than in others. Furthermore, all measures have weaknesses (some more so than 
others) and where one is weak another measuring a similar variable may be strong, 
and vise versa. Therefore, whenever it is feasible, it is advised to use 
multiple measures and indicators for a particular learning outcome unless one 
has complete confidence in one of the measures. If they all indicate the same 
thing, one's assurance that the indication is actuality is greatly increased. 
As an example of how multiple measures lead to greater reliability and validity, 
let us take the case of unobtrusive measures. Thelin (1977) has emphasized that 
unobstrusive measures "have to be considered in clusters and tied to a conceptual 
framework if they are to be of significance for institutional monitoring [p. 133]." 
Unobstrusive measures--for example, an increase in attendance at campus plays and 
art displays after the occurrence of a demonstration program on appreciation of the 
arts sponsored by the Office of Student Affairs in the dormitories— can be quite 
useful and revealing measures. See the book by Webb and associates (1966) for 
helpful, in-depth discussion about such measures. 

Many times it is possible to use data that have been collected for other 
purposes (for example, data from student transcripts, administrative files, and 
community records), which some have called "secondary data." Usually we think 
immediately of having to collect data when there is need for an assessment study, 
and yet we might be able to get by without having to go to the trouble of collecting 
data for the study. Although they save costs and time, however, in addition to 
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providing useful supplemental evidence, such data can 1ea~ to serious problems if 
great care is not taken in using them. Boyd and Westphall (1972) provide 
criteria for determining when particular secondary data are acceptable for a 
particular situation and use, plus they discuss precautions for avoiding the 
potential pitfalls. 

Those doing assessments of various kinds have traditionally limited 
themselves to paper-and-pencil tests, questionnaires, and interviews for 
data collection, even though a variety of other methods have been shown to 
be practical, valid, reliable, and cost effective for particular purposes 
and contexts. Lenning (1978a) found fifti different data collection methods 
in the literature that were being recommended for assessments of various 
kinds. Knapp and Sharon (1975) review a number of these methods, which can 
be used instead of or to supplement the traditional data collection methods. 
As was true for indicators and measures, and for the same reasons, the use of 
multiple data collection methods is desirable whenever it is feasible. That this 
can be feasible and cost effective was shown by the learning assessment system 
developed and implemented at Empire State College (Palola and Lehmann, 1976). 
They supplement standardized and local test score data with student self report, 
instructor observations, writing samples, and administrator observations. 

Collecting Data . One can have exactly the measures and data collection methods 
needed for collecting data, and yet it all can be for naught if one doesn't 
plan well (concerning such things as who to collect the various data from, how to 
approach them, and sampling procedures) and use care in the actual collection of 
the data. For example, a poorly worded cover letter with a questionnaire could 
easily cut the response ratt in half or more; so could administering the 
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I questionnaire shortly before mid-;erm exams. Much time, money, 

and frustration can be saved if one takes pains to have: well -designed 
J interview forms, written instructions to read for test administrators, careful 

I selection of samples, questionnaire items free of bias, wel 1 -designed 

pilot tests to try out procedures ahead of time, procedures for maximizing 
I response rate (such as showing the need for such data, and promising and giving 

m respondents feedback a^out the results), sensible coding and data formatting rules, 

■ careful editing procedures, etc. The appendices of Micek, Service, and Lee 
I (1975) provide many helpful suggestions in this area, 

I Using the Data: analysis, interpre tat ion, reporting, and application . Most assess- 

■ ment studies rely exclusively on the use of simple descriptive statistics such as 
means, standard deviations, and tabulations and cross-tabulations of freanencies and 

I Dercentaqes. Much useful information can be obtained from the use of such simple 

statistics, especially if they are profiled graphically and patterns of similarities 
I and discrepancies across information items and across qroups are examined. Means by 

m themselves can be quite misleadinq if the frequency distributions are not also 

examined. Also, response bias should be analyzed in questionnaire and interview studies. 
I It is often useful to make comparisons across groups when group difference 

on other characteristics (for example, input variables) are taken into 
I consideration. Although "eyeballing" across groups can be revealing, it should 

I b2 supplemented with statistical tests such as t-tests, chi square, analysis of 

variance, correlational analysis, and discriminant analysis. Unless the staff 
I member is experienced in such techniques, however, the aid of a statistician 

on campus should be obtained in planning the analysis. In planning the study, 
I the staff member and the analytical design expert should consider using one of 

■ the analytical designs proposed by Campbell and Stanley (1963), and many helpful 



resources that discuss the selection and use of statistical methods are 
available (for example, Tatsuoka and Tiedeman [1963] and Siege I [1956]) • 

In outcome studies, change in status is often of concern. However, most 
analysts now agree that change scores or average change should not be used 
in such analyses. Rather, they advise comparing the final status of the 
students to the final status of other students having the same initial ability. 
For comparison across groups this can be accomplished by random appointment 
to each group initially, by group assignment Ihrough stratified random paired- 
matching on input level, by comparing across similar initial -level strata, or 
through the use of sophisticated statistical adjustments to post-test scores 
that effectively equate initial levels (for example, analysis of covariance). 

Interpretation and use of data are also crucial elements in an assessment 
study, and too often the application of such results is ineffective. If the 
assessment data are to have any impact, the users of the data must be precisely 
identified early in the assessment planning process, prior to conducting the 
study. Input should be solicited from them concerning their specific concerns 
and what assessment information would be helpful to them in their decision 
making. Such input will serve a primary role in determining what study groups, data, 
and analyses are desired for the study. Once analyses are completed, brief, 
concise reports tailored to each person's informational needs should be sent 
to them. Graphical presentation can often be helpful in such reports. One 
interesting and potentially useful way of making these reports attention-getting 
is through a "peer group--intergroup" model proposed by Alderfer and Holbrock 
(1973) and used by Hecht (1977). Selected college staff prepare "action-oriented" 
written and oral responses to the evaluation data for presentation to other 
college staff at their levels , 
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Special Considerations for Different Kinds of Assessment 

The general steps outlined in the preceding section apply to both the assess- 
ment of groups and the assessment of individuals. For the assessment of individuals, 
however, they apply in a much more informal, subjective way than is true of 
assessment of groups. Furthermore, an important difference is that for assessment 
of individuals the instruments used must have much hiqher reliability coefficients 
(in the 0.8-0.9 range versus as low as 0.6 for groups) tc be useful. 

Another dimension on which there are differential considerations is the 
assessment of students versus the assessment of nonstudents. Usually one is 
interested in assessing different factors for students than for nonstudents. Secondl 
the assessment of students often involves the use of standardized paper-and-pencil 
instruments, such as achievement and ability tests, whereas most non-student 
assessments of interest to student affairs workers will not make use of such 
instruments. To illustrate, May (1975) discusses two types of staff member 
assessment (he cal'is it evaluation) for the guidance setting: 
For assessment of competency (p. ): 

Each individual counselor lists the ten or twelve functions 
that must be achieved to have a quality guidance program. These 
are services that you feel so strongly about that you would defend 
them before a group of district administrators or before the 
school board. Place thi l ^ist in rank order . . The combined staff 
list will constitute the values for the guidance program , , , At 
the conclusion of the year, prepare a short narrative explaining 
how you achieved, or why you failed to achieve, each goal and 
request your supervisor to do the same. Exchange narratives, 
wait several days and meet to discuss differences, 

* 

For technical competency assessment (p, ): 

To be useful, evaluation of process must provide a direct 
measure of behavior change or knowledge learned as a result of 
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contact with the counselor. Perhaps the following approach could 
be used. 

1. Detail each task, formulate specific behavioral objectives 

and write implementation strategies. 

2. Describe what a student who has achieved the specific 

objectives does that distinguishes that student from one 
who has not achieved the objectives. 

3. Outline precise situations that require the learners 

to demonstrate ••'lether they can react in a knowledgeable way. 

4. Develop a pre-determined method for recording and evaluating 
responses to situations. 

These are two examples of individual staff members assessing themselves 
and being assessed by others. There are probably many other ways in which 
such assessment could be carried out. 

When it comes to assessment of individual students, student affairs 

workers (and especially personal counselors) have been considered experts for 

many years. However, the traditional focus on student affairs personnel doing 

the assessment of students is being replaced in many quarters by student affairs 

personnel helping students to assess themselves. As outlined by Miller and 

Prince (1976), the goal of assessment for student development "is to help 

students understand thpi'r current patterns of behavior, emphasizing positively 

the specific skills they have instead of the ones they lack. From this base, 

all students can move toward increased self-direction . . . Assessment programs 

must be designed with students rather than for or about them; therefore, only 

information that can directly increase students' self-understanding or improve 

their self-direction need be collected. The primary focus of many student 

assessment efforts has been to help student affairs workers better understand 

their 'clients.' Although this objective is desirable, it has tended to 

create volumes of information about stud-.nts that is rarely used directly by 

them" (pp. 48-49). t ~ 0 
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Miller and Prince have much good discussion focusing on how to conduct 
such self assessment. They do not limit this "assessment for student develop- 
ment" to individuals, however. They define it as "the process through which 
students , groups , and organizations systematically acquire and use data from a 
variety of sources to describe, appraise, and mod'^y their own development . Thus 
this method "differs from the more traditional approaches in its purpose, in what 
is assessed, in the techniques used, in the way it is implemented, and in the 
role the student affairs practitioner plays." (p. 47) 

A word should also perhaps be said about another noteworthy assessment 
differentiation. Assessment of certain types of factors have become separate 
areas of specialization. A good example is "needs assessment," where discrepancy 
between "what is" and "what should be" is a primary focus. Also, needs assess- 
ment is noteworthy in that there is a serious definitional problem among 
assessors concerning what is a "need" (Lenning, 1978b; Lenning and McAleenan, 
in press). Other areas of specialization within assessment include assessment of 
ability, achievement, personality, goals, values, interpersonal functioning, 
and organizational functioning. 

Strategies for Evaluating 
Student Service Activities and Programs 

In their excellent book of readings and comment pertaining to the theory 

and practice in educational evaluation, Worthen and Sanders (1973) open with a 

serious charge that unfortunately seems to still be true: 

Evaluation is one of the most widely discussed but little 

used processes in today's educational systems. This statement 

may seem strange in the present social context where attempts to 

make educational systems accountable to their publics are proliferating 

at a rapid pace . . . yet, despite these trends toward accountability, 

only a tiny fraction of the educational programs operating at any 



level have been evaluated in any but the most cursory fashion, if 
indeed at all. Verbal statements about evaluation and accountability? 
An abundance. Genuine evaluation of educational programs? Unfortunately 
rare. (p. 1) 

This statement applys to student affairs programs as much if not more than 
other areas of education, especially the second part which indicates that 
programs "have been evaluated in any but the most cursory fashion, if indeed at 
all. M Whereas most instructional programs have a large number of clear-cut 
learning go that are relatively easy to state in measureable outcome 
terms, student affairs goals for students often tend to be imprecise, vague, 
and illusory in nature, and difficult to measure. This is because they are 
more often emphasizing affective development while formal instructional programs 
are more often emphasizing cognitive development. Secondly, they are more 
often emphasizing a far wider array of goals for students; which could mean 
that they are spreading their focus too thin. Counselors and other student affairs 
people also perhaps have more of an aversion to empirical data and analysis than 
do academicians involved primarily in research and scholarship activities. 

On the other hand, being seen to serve a support rather than a line role 
in most postsecondary institutions, student services people have increasingly 
started to recognize their vulnerability in any projected financial cutback and 
retrencnment. For their programs to re in support, student services personnel 
must do a more effective and concrete job of communicating to others the important 
and central benefits students can gain from their programs, and must provide 
factual evidence of such benefits occurring. Furthermore, they must demonstrate 
that their activities and programs are both efficient and effective. Only through 
effective program evaluation can such evidence be developed, .and such evaluation 
must begin with a clear and concrete delineation of activity/program goals and 
objectives. 
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Burck and Peterson (1975), with "tongue-in-cheek cynicism, " discuss 
seven of the most common evaluation strategies, or models, used in student per- 
sonnel services. All of them are really not program evaluation at all, according 
to Burck and Peterson, These "models" are: (1) the Sample-of-One Method, which 
involves discussing the problem with one or two colleaques and arriving at a 
consensus; (2) the Brand A versus the Brand X Method, which compares nonequiva- 
lent " ipples-versus-oranges" groups; (3) the Sunshine Method which solely provides 
eviden:e of program quantity and extensive client exposure to the program; (4) 
the Goodness-of-Fit Method, which establishes how standard the program is and 
how well it fits into established procedures; (5) the Committee Method, which 
involves a group of people connected with the program discussing and reaching 
consensus on the program's effectiveness and writing a report for those in 
authority that points out and extolls its merits; (6) the Shot-in-the-Dark 
Method, where clear proqram objectives are lacking for the program, or where 
evaluation is entirely divorced from program goals, so evaluation involves a 
random search for any kind of an impact that might be possible; and (7) the 
Anointing-by-Authority Method, where nationally recognized con c ants are 
brought in to confirm pre-ordained findings through talking with the "right 
people," and whose name and status legitimize those findings in the eyes of 
those in authority. The one thing that all of these common (and they are common) 
approaches have, in addition to being poor evaluation, is that none of them 
objectively try to determine the real outcomes of the program. 

Another author who has discussed the need for better evaluation in student 
services work, and specifically in counseling, is Krumboltz (1974). Rather than 
talk in terms of evaluation, however, he focuses on accountability; and presents 
a number of benefits to counselors from an effective accountability model. 
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Similar benefits could accrue to other areas of student affairs if a good 
evaluation system were implemented. His proposed benefits are as follows 
(pp. 639-640)-- 

.. .would enable counselors to: obtain feedback on the results 
of their work; select counseling methods on the basis of demon- 
strated success; identify students with unmet needs; devise 
shortcuts for routine operations; argue for increased staffing 
to reach attainable goals; (and) request training for problems 

requiring new competencies By learning how to help clients 

more effectively and efficiently, counselors would obtain: more 
public recognition for accomplishments; increased financial support; 
becter working relationships with teachers and administrators; 
acknowledged professional standing; (and) the satisfaction of 
performing a constantly improving and valued service. 

Krumboltz does not present such a model, but he does propose seven criteria 

for such a model, or system, that are worthy of attention and adoption (pp. 640- 

641 ) : 

1. In order to define the domain of counselor responsibility, the 
general goals of counseling must be agreed to by all concerned 
parties . 

2. Counselors 1 accomplish ents must be stated in terms of important 
observable behavior changes by clients. 

3. Activities of the counselor must be stated as costs, not 
accompl ishments. 

4. The accountability system must be constructed to promote professional 
effectiveness and self -improvement, not to cast blame or punish poor 
performance . 

5. In order to promote accurate reporting, reports of failures and 
unknown outcomes must be permitted and never punished. 

6. All users of the accountability system must be represented in 
designing it. 

7. The accountability system itself must be subject to evaluation and 
modification . 

A number of strategies, or models, have been developed by evaluation theorists 
and practitioners in the area of curriculum development that provide viable alterna- 
tives to meet distinct conditions and situations, and that have aspects or 
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components that student affairs program evl auditors should consider applying. 
Worthen and Sanders (1973) reprinted original writings by a number of model 
developers that they felt had made important contributions to evaluation practice 
and could provide frameworks for such practice. Following each reprint, they 
separately discuss each model in terms of potentials and limitations, and then use a 
chart (pages 209-220) to compare them on 12 factors or dimensions: 
(1) definition, (2) purpose, (3) key emphasis, (4) role of the evaluator, (5) 
relationship to objectives, (6) relationship to decision-making, (7) types of 
evaluation, (8) constructs proposed, (9) criteria for judging evaluation 
studies, (10) implications for evaluation design, (11) contributions to the 
design of evaluation studies, and (12) limitations and possible misuses of 
the approach. 

Contributions that these and other models have to make to an overall frame- 
work of viable concepts and alternative strategies and procedures from which one 
can choose for the evaluation of student services wil'i be outlined here. As Scriven 
suggests in his thoughtful article (1967) y the major focus in the models may be on 
curriculum evlauation, but the same ideas can also be applied to other kinds of 
evaluation, such as student services evaluation. A special focus of the following 
discussion will be on the procedures for implementing each model, which can suggest 
evaluator skills and competencies needed for such implementation. After a brief 
discussion of noteworthy and relevant points from each model, as perceived by this 
writer, the discussion will conclude with categorizations of model types proposed 
by two authors who attempted to place the various models into perspective. 

The Scriven Formulations 

Scriven (1967, 1971) made a number of useful points. First, he 
differentiates the goals of the evaluation from its roles. The goals may be 
usefully formulated in terms of questions regarding performance, worth, merits, 
drawbacks, etc. The goals lead to "gathering and combining performance data 
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with a weighted set of critical scales to yield either comparative or numerical 
ratings, and in justification of (a) the data-gathet ing instruments, (b) the 
weighxings and (c) the selection of criteria [p. 40]. M Wh^le the goal i n essence 
focuses on the gathering of valid and reliable evidence relating to the worth 
of the service or activity, the role being performed can vary widely. 

Two basic roles that Scriven describes are the formative role of how to 
improve the service or activity, and their outcomes on students or others, 
and the summative role of whether the service or activity should be terminated, 
continued as is, or continued in a modified form. Some of the same data may 
serve both roles, but the two roles mainly require different types of data- 
in the first case data that assist understanding and in the second case data 
that assist value judgement. Scriven feels that evaluators performing the 
formative role should be regular members of the on-going program staff and 
that the summative role should be carried out by unbiased, independent, outside 
people without a vested interest in seeing that the program SMCceeds (it Joesn't 
have to be someone outside of the institution, just someone outside the program 
who ,i?s no vested interests in the program), and who can thus be objective and 
master of fact about the matter. 

An additional role differentiation for evaluator* mentioned by Scriven (1971) 
was (1) evaluating the goals of the program, versus (2) evaluating whether and how 
well the program goals have been achieved. Formal program evaluation studies have 
almost always focused on judging the worth of the program a k J/or how to improve 
the urogram in terms of the qoals of the program developers, without considering 
the ;rth of the goils. Scriven r >ays, however, that an evaluator should also 
evaluate t- a appropriateness and worth o* the program goals. Furthermore, he 
indicates that the goals should be evaluated prior to evaluating the attainment 
of those goals; if it is not a good goal it really does not ma. *er whether or not 
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B the goal has been achieved. Evaluating the appropriateness of goals involves 

■ agreed upon, objective criteria for what constitutes a good goal, plus applying 
logic. Unlike uhe evaluation of goal achievement, goal evaluation does not 

I involve measurement; goals cannot be measured as can achievement. According 

- to Scriven, the welfare of the consumer and society should be important criteria 

m in evaluating goals. 

I Scriven made one additional noteworthy point concerning program goals—that 

the evaluator should not focus so intently on whether the program goals have been 
I achieved that he/she fails to notir* significant non-intended program outcomes 

■ that have taken place. According to Scriven, one should definitely look for 
evidence of unintended outcomes and be open to considering equally such evidence 

I with evidence of accomplishment or lack of accomplishment pertaining to the program 

goals. The importance of this admonition is demonstrated by Lenning and associates' 
I (1977) finding that unintended outcomes can be positive as well as negative, and 

■ can have as much or more evaluative impact on program planning as the intended 
program outcomes. Even negative outcomes, which many people wish to ignore, should 

I be considered because they can suggest important modifications that are needed in 

the program. 

■ In reaction against the commonly accepted strategy of devoting all evaluative 

■ attention to the outcomes intended or planned for the program, Scrivpn (1972) 
developed a model that looks for significant program impacts of any kind, whether 

I they are implied by the program goals or not, which he called "goals-free evaluation." 

Scriven* s idea is that an outside evaluator sf.ould be brought in and should deducr 
I what appears to be the programs 1 goals based on his/her observations. Cnly then 

■ should the evaluator talk to the program staff about their intended program goals. 
There are problems with goals-free evaluation as a method, but Scriven's point is 

I well taken. Furthermore, Scriven does not downgrade the importance of program 

goals for evaluation, as the following quote (Scriven, 1971) indicates: "The 

■ statement of goal narrows our problem to manageable size. We can't apply all 
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possible tests to every sample in order to look for all possible effects. We 
check in the general area where the shot was aimed, keeping our eyes open for 
any side effects." 

Another useful Scriven formulation is his distinction between what he calls 

"secondary " or "intrinsic" evaluation and "primary" or "pay-off" evaluation, 

evaluation of the means used to bring about desired end results versus evaluation 

of those end results. He suggests that secondary evaluation is fine if we can make a 

solid link between the secondary indicator and the primary payo.f; otherwise 
primary evaluation is essential. However v by doing both types of evaluation in 

the same evaluation study understanding is increased and we have greater 

assurance that our judgements of worth are reliable. Intrinsic and pay-off 

evaluation both can be applied in formative evaluation as well as in summative 

evaluation . 

Reference should also be made to a general five-step procedure 
for conducting progrvn evaluation that Scriven outlined (1971): (1) identify and 
assess the intended goals of the project or program, using ratings of such things 
as social utility, necessity for the goals at this point in time, and the number 
of people that will be benefited if the goals are met; (2) measure the program's 
effectiveness, where effectiveness is not restricted to the stated or implied 
goals; (3) assess program costs (here Scriven uses a 12-point check list and 
breakdown involving installation versus maintenance costs, dollar versus psychic 
costs, per-stude t versus per-system costs, etc. (4) ajssess the program availability 
and practicality; and (5) produce an overall summary report. Scriven was talking 
to professional evaluators when he presented this list, but it is also relevant 
for other evaluators. 

In closing this discussion of Scriven's formulations, several additional 
noteworthy points should be made. One is Scriven 1 s contention that process 
research, research aimed at describing and understanding the processes taking place 
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within a service program, only becomes evaluation if its focus is on judging 
the merit or worth of the process or on improving the process. Thus, the common 
view of accountability illustrated below by Wickline (1971) is clearly not program 
evaluation, in spite of what many people might think. 

If one sat in on Congressional hearings today, he would find 
that, in justifying the expenditures in education, people 
still talk primarily about how the money has been spent, what 
kind of materials and equipment have been purchased, the number 
of children who have been served and the number of teachers who 
have been involved. They talk very little about what has been 
accompl ished. 

On the other hand, Scriven does not ignore process, nor does he ignore 
costs. He contends that data about program benefits should be related both to 
data about program process and program costs. And when it comes to costs and 
benefits, Scriven feels that the costs and benefits of the evaluation process 
itself should also be explored, preferably in probable terms pnor to the time 
that the evaluation is to commerce, and the evaluation process modified accordingly. 

Finally Scriven demonstrates the desirability of using comparison groups, even 
though we cannot control for particular input characteristics that could 
account: for differences in findings among groups, and even though the differences 
expected are usually small. He also discusses the usefulness of taxonomies 
in specifying in more detail the criteria of achievement or other outcomes to 
be used in an evaluation study, and he proposes use of his modified version 
of Bloom's (1956) taxonomy. Perhaps more useful for such purposes 
in student affairs evaluations *s the comprehensive taxonomy of postsecondary 
education "types of outcomes 11 and "audiences" developed by Lenning et a] (1977) 
because it focuses much more in noncognitive areas than does 'Scriven 1 s. That 
taxonomy is oi ,4, lined in Figures 1 and 2. (Standard definitions and sample 
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Figures 1 and 2 go about here 



measures or indicators are provided by Lenning et al [1977] for all categories 
and subcategories of Figure 1.) It should be useful to also apply Scriven's 
taxonomy, however, to generate student service evaluation coverage ideas. 

Stake's Countenance or Preordinate Evaluation 

Stake (1967) emphasized how informal, subjective evaluation differs from 
formal, objective evaluation. Informal evaluation depends on "casual observation, 
implicit goals, intuitive norms, and subjective judgment," while formal evaluation 
depends on "check-lists, structured visitation by peers, controlled comparisons, 
and standardized testing of students [p. 523]." Informal evaluation ^s what 
usually takes place, and can provide effective and penetrating insight. However, 
it can as often be superficial and distorted, so Stake suggested that rational 
judgment requires the use of formal evaluation. He suqgested that description and 
judgment are tne two major activities in program evaluation, and that they should 
be many-faceted and comprehensive activities focusing on all aspects of the program. 

This Model distinguishes among three types of data that are necessary 
for hoth description and judgment, antecedent data (input conditions and 
characteristics not part of the program that may relate to outc les), transaction, 
data (interactions and encounters between the clients and program [including 
program personnel] that constitute the program processes), and outcome data 
(short- a. id long-term consequences of the antecedents and transactions on the 
clients, the program and its components, and other persons or entities). A 
matrix with twelve cells describes the data that should be collected for the 
evaluation, with antecedents, transactions, and ojtcomes forming the vertical 
axis and intents (goals and objectives), observations of outcomes, standards 




Figure 1 

TYPES OF OUTCOMES THAT COULD BE AIMED FOR 
IN PCSTSECONDARY EDUCATION 3 



Gregory 
Code Number 



Entity Being Maintained or Changed 



1000 ECONOMIC OUTCOMES 

1 100 Economic Access and Independence Outcomes 
1110 Economic Accres 

1120 Econoihir Fk*.b.l.tv. Addpt*l>m.y. and Security 
1 1 30 Income and Standard of Living 

1200 Economic Resources and Costs 

1210 Economic Coots ann Efficiency 
1220 Economic Resource* including employees) 

1300 Economic Production 

13iu Economic Productivity and Production 
1320 Economic Services Proviucd 



Category 
Code Number 



Entity Being Maintained or Cnanged 



1400 Other Economic Outcomes 



2000 HUMAti CHARACTERISTIC OUTCOMES (continued) 
2760 Power and/or Authority 

27/U Jot). School, or Life Success , 
2780 Other Statu*. Recognition, and Certification Outcome j 

2800 Social Activities and Holes 

2810 Adjustment to Retirement 

2820 AMihations 

28S0 Avocational and Social Activities and Roles 

2040 Career jryj Vocational Activities and Roles 

2650 Citizenship Activities and Roies 

28CC Family Activities *ind Roies 

?fl70 Friendships «nd rieictionbhips 

2&fc0 Ottier Activity and Role Outcomes 

2900 Other Human Characteristic Outcomes 



I 2000 HUMAN CHARACTERISTIC OUTCOMES 



2100 Aspirations 

2110 Desires, Aims, and Goals 
2;20 Oiblikes, Likes, and interests 
2130 Motivation or Drive Level 
2U0 Other Aspirational Outcomes 

2200 Competence and Skills 
2210 Acaoemic S> ills 

Citijcnshipjrid Family Membership Skills 
Cicititity tikillb 

Fxpression J«'l Communication Skills 
Intellectual b.kihs ... ... 

Interpersonal, t e^Orship. ana Organizational Skills 
Occupational ano trnpiuyabtiity Skills 
Physieai and Moior Skills 
Other Skill Outcomes 



2220 
22 \0 
2240 
2250 
2260 
2270 
2260 
2290 

2300 Morale, r-.Msfoction. and Affective Chdracteristics 
2310 Alt ides and Values h . lita 

2320 Belitis, Commitments, and Philoscphv 01 Lite 
23j0 Feelings and Emotions 
2340 Mores, Customs, and Standards of Conduct 
2350 Other Affective Outcomes 

2400 Perceptual Characteristics 

2410 Perceptual Awareness and Sensitivity 
2420 Perception of Self 
2430 Perception of Others 
2440 Perception of Things 
2450 Other Perceptual Outcomes 



2500 



2000 



Personality and Personal Coping Characteristics 

2510 AUvcnturousness and Initiative 

2520 Autonomy and inceoendonce 

2530 Dependability and Responsiomiy 

2540 Dogmatic/Open. Minded. Au'hor.tdr.an/Democratic 

2550 F legibility and Adaptability 

25C0 Habits 

25.'0 Psychological Functioning 

2530 Tolerance and Persi3tC'nce 

2590 Other Personality and Personal Coping Outcomes 

Physical and Physiological Characteristics 

2610 Physical Fitness ana Traits 

2620 Physiological Health 

2630 Other Physical or Physiological Outcomes 



2700 Status, RccoQniliun. and Certification 

2710 Completion or Achievement Av*ard 
2720 Credit Recognition 
2730 Image, Reputation, or Status 
2740 LicenSirc and Ci-rtitication 
2750 Obtaining a Joo or Admission to a Follow-up Program 



3000 KNOWLEDGE, TECHNOLOGY, ANO ART FORM OUTCOMES 

3100 General Knowledge and Understanding 

3110 Knowledge end Understanding of General Facts and 
Terminology 

3120 Knowledge and UM rstanding of General Processes 
3130 Knowledge and Undemanding of General Theory 
3140 Other General Knowledge and Uncu'Siandmg 
3200 S<>LCi?'i2Cd Knowledge arr Understanding 

3210 Knowledge and Understanding of bpecialized r-acis 

and Terminology 
3220 Knowledge and Undemanding of Specialised 
Processus 

3230 Knowledge «mu Understanding of Specialned Theory 
3240 Other Specialized Knowledge and Understanding 

3300 Research and Scholarship 

3310 Research and bchoiarshio Knowledge and 

Understanding 
3320 Research and Scholarship Products 

3400 Art Forms and WorKS 
3410 Architecture 
3420 Dance 
3430 Debate and Oratory 
3440 Drama 

3450 Literature and V'riting 
34GO Music 

3470 Painting Drawing, and Photography 
3480 Sculpture 
3490 Other Fine Arts 
3500 Other Knowledge. Technology, and Arl F orm Outcome's 

4000 RESOURCE AND SERVICE PROVISION OUTCOMEZ 

4100 Provision of Facilities and Even.s 
4110 Provision of Facilities 
4120 Provision or Sponsoiship of Events 

421K) Provision cf Direct Services 
4210 Teaching 

4220 Advisory and Analytic Assistance 
4230 Treatment, Care, and Referral Services 
4240 Provision of Othe Services 
4300 Other P source and Service Provision Ouicomos 



5000 OTHER MAINTENANCE AND CHANGE OU1COMES 
5106 Aesthetic-Cultural Activities, Traditions, and Conditions 
5200 Organisational Format. Activity, and Operation 
5300 Other Maintenance and Change 



-From the type-of-outcome dimension of the NCHEMS Outcomes Structure (Lenning et al 
1977? p. 27) The Structure provides standard definitions and sample measures or 

indicators for each category, along with ^ocedures for subdividing ,nto additional 

levels of categories. 100 

178 -tod 



BEST COPY AVAILABLE 

Figure 2 



PEOPLE OR ENTITIES THAT MAY BE RECIPIENTS 
OF VARIOUS POSTSECONDARY EDUCATION OUTCOMES 3 

10 mutual/Group Cfienfs-Thla category refers » arsons or groups or persons who are direct clients of the poslsecondary education unit of 
concern and/oi tncir Immediate ^socldtes. such as family and relatives or poors 

11 Srudenls-tndlvtduala cr groups ot indiv.dup' who currently are enrolled In theprogtam, institution, or system of poslsecondary education 

12 former Studon ts -Individual* or groups 3l Indiv.du afs who formerly were enrulled in the program, Institution, or system of poslsecondary 
education 

13 Family and Relatives ot Student i or Former Station ts 

14 Peers and^ssocrafeso/ Students or format Students 

15 Faculty 

1C Statt Other than Faculty 

1 7 Other /nawdut //Croup C/ienis-An example would bo an Individual whe 'a none ot the above Out is sawed by an advisory sorv.ee offered 
Oythe cuhtge 

20 interested Communities ca egory r.lers to large groups that art Identll.ed as entities working toward a wlhdetined interest or 
mission 

21 Private Enterprise Communities- Communities where a major purpose is financial remuneration and protit-for example, corporations. 
Smalt businesses, arid farmers 

22 a.ssoci*»on Commur»l.*s-Cammun,ti#s where members belong on the basis of affiliation rather thin employment, such as unions and 
profc5Mur>«il socictitv. 

23 Govevnmenr Commutes -Communis designed to aUm.n.sler government regulations and services, such as city hall, stale department 
of education, and legislative communities 

24 Nonoovrrr,menfe//PuO/ 1 c Sendee Communis Oflcr thtn the Institution Producing the Outcome-Nonprofit service conizations, such 
T£oo^ wnc-s. Philanthropic founcat.ons. colleges tome, l.un iro college producing .he outcome), and research 
organizations 

25 rVisrrfulronor Institution'! Un,t Producing the Ou/come-The poslsecondary education institution and/or units w.tn.n thai nstilutton that 
are pcice»ed as the Produce / faultUtor ol the outcomoui of concern 

26 Other interest- Based Communit.es -An example would be an ad hoc coalition task fo.ee of representatives from two or inoro ol .no uuove 
areas 

30 Geographic 8j$eo Commumf/es-Th.s category refers to large groups defined on the bas.s of functional ferritin* boundaries 

31 Local Cc*munm/-A township, c.ly. county metropolitan area or other type of loca;.t, nav.ng pan.cjiar boundar.es II .s not « ^Msariiy 
resisted to tne legal oi jurisdictional boundary Out ire functional one .n wmch tru .mpact of mo .nstilunon .s <or Should be) d.rectiy ana 
phys'catly Irlt. The boundaries w»n vary *im the insutunon/piogiani ana outcome of conewn 

32 The Staie 

33 A Region— In aggregation of states or parts of states 

34 The Nal'On 

35 An International Community 

36 Other Geographic Based Communities-^ . ample would be a lesearch o.scovcry that a'fecls primarily people living in me coldest 
latitudes, or where ■( sr.o«* heavily 

40 Aggregates ol People -Th,s category refers to subpopuiat.ons of oeopie durmguished by porucut.. characteristics that may .ndicata comnon 
concerns, needs o?wa/>fa. but wno 00 not necessary have a common interest or mission, and therefore oo r ot constitute communities 

41 Ability Level SuOponiW«>ns-$ubpopui*lion, del, od according to lev* of ability, prof.c.er.cy on general intellectual functioning or 
specific >k.ils-tor example, gifted typical, disadvantaged, or skilled, semi Skilled >• skilled 

42 Ag§ SuOpopulations 

43 Boucationjt Level SubpopuHtions 

44 Income l evei Suopopulations 

45 Occupation Subpopulations 

46 Physical Diseoihty Condition Subpooutations 

47 Race Svbpopuiations 
40 Sen Subpopuletmns 
49 Other Stch Agyregetes 

50 Other Avowees — Examples wruid te ihe natural env.ronmont that -s affeeicJ by university <non$o.uO rASnarch (wh.cn ,n turn would be 
expected to tMve .mpacts on jt>o.enL.»s such as individuals and con,, lun.l.es) ana popwiatiuns ol an.mati (such as ma animals alfected by 
effurts to t-uep d«>|iKitud SHk.iv1 limn twcommy lutlitcl ui t»r 0»'>^» • "onl Of »olonnj'y nieUK .no ,» 
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a Frora the "audience" dimension of the OEMS Outcomes Structure (T.enninp ct_nl , 
1977. |>. 24). T/9 ^ 84 



that the clients expect, and judges' perceptions of program value forming 
the horizontal axis. The maorix, which is shown in Finure 3, is vertically 
split into a description section and a judgment section. 

Figure 3 goes about here 

Concerning the d escription section of the matrix, intents includes program 
effects that are planned for or intended, desired, hoped for, anticipated, and 
those that are feared. A major problem for the evaluator is to deduce the 
intentions for the pror and transform them into concrete, useable data. Obser- 
vations include descriptions of surroundings, events, behaviors, and consequences 
that are gathered through direct observation and with the help of instruments such 
as "inventory schedules, biographical data sheets, interview routines, check lists, 
opinionnaires, and all kinds of psychometric tests." 

During the analysis phase, the focus is on discovering r * each row of the 
matrix how much of what was intended actually happened (discrepancies between intents 
and observed occurrance, and the amount of congruence), and also on examining the 
relationships (or contingencies) among antecedents, transactions, and outcomes 
across both the intended and observed dimensions. In addition to the congruence and 
contingency comparisons of descriptive data that have already been discussed, 
Stake stresses the importance of examining whether the ; rogram plan (intents) 
logically relates or conforms to the philosophic background and basic purposes of 
the program; whether the intents constitute a plausible and well -thought out plan 
for implementing the program rationale (philosophy and purposes). 

Concerning the judgment section of the matrix, Stake indicated that a 
wholistic view is imperative but not being taken in most current evaluation 
efforts within education. As he stated it, "it is a great misfortune that 
the best trained evaluators have been looking at education with a microscope 
rather than with a panoramic view finder." 



Figure 3 

CATEGORIES OF DATA NEEDED FOR COMPREHENSIVE 
HROGRAM EVALUATION ACCORDING TO STAKE'S COUNTENANCE MGDEL 



DESCRIPTION 



Intents 



Observations 



A£t 

t 



ecedents 



nsactions 



comes 



JUDGMENT 



Standards 



Judgments 



Antecedents 



Transactions 



Outcomes 
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The descriptive data about the program, discussed earlier, can be judged 
with respect to sets of absolute standards or criteria of what the antecedent, 
transactional, and outcome intensions and observations should be (absolute 
comparison), each set conforming to the view of a relevant reference group or 
point of view. The other possibility is to judge the data with respect to 
the same data for similar programs elsewhere (relative comparison). In doing 
the judging, the evaluator assigns a weight of importance to each set of 
standards he/she considers to have relevance, and determines the urogram data on 
which to make comparisons. Based on relative and absolute judgment, the evaluator 
arrives at "an overall or composite rating of merit 11 and specifies limitations 
and qualifications that apply to this rating. He/she then develops recommendations 
r arding the program decisions of concern (continue the program as is, terminate 
t , program, modify the program in particular ways). 

Stake (1967) made it clear that all of what he proposed does not have to be 
done in one massive effort. Furthermore, because of available evaluation 
resources, many evaluation efforts will devote all of their resources to only 
certain areas of programs and to particular types of data (for example, intended 
versus actual outcomes). And Stake makes it clear that the process for narrowing 
down the focus should be deliberate and formal also, as indicated below: 

Educators should be making their own evaluations more 
deliberate, more formal. Tnose who wil l--whether in their 
classrooms or on national panels— can hope to clarify their 
responsibility by answering each of the following questions: 
(1) Is this evaluation to be primarily descriptive, primarily 
judgmental, or both descriptive and judgmental? (2) Is this 
evaluation to emphasize the antecedent conditions , 'the trans- 
actions, or the outcomes alone, or a combination of these, or 
their functional contingencies: (3) Is this evaluation to 
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indicate the congruence between what is intended and what 

occurs? (4) Is this evaluation to be undertaken within a 

single program or as a comparison hetween two or more 
curricuiar programs? (5) Is this evaluation intended more 

to further the development of curricula or to help choose 
among available curricula? With these questions answered, 
the restrictive effects of incomplete guidelines and in- 
appropriate countenances are more easily avoided, (p. 540) 

Stake's Responsive Evaluation 

In later years, Stake changed his position about the relative value of 

formal and informal evaluation (1973). Whereas, he previously downgraded 

informal evaluation, he no favors it in the majority of cases because it is 

performing more of a service. Thus, he titles his new model "resoonsive evaluation. 

Responsive evaluation sacrifices some, measurement precision, but aains in 

the value of the findings to those involved with the program. According to 

Stake: 

Responsive evaluation is le s reliant on formal communication, 
more reliant on natural communication. . . It is evaluation 
based on what people do naturally to evaluate things: they 
observe and react. . . Subjectivity can be reduced by repli- 
cation and operational definition of ambiguous terms even 
while we are relying heavily on the insights of personal 
observation. An educational evaluation is responsive eval- 
uation (1) if it orients more directly to program activities 
than to proqram intents, (2) if it responds to audience 
requirements for information, and (3) if the different value- 
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perspectives of the people at hand are referred to in 
reporting the success and failure of the program. ... To do 
a responsive evaluation, the evaluator of course does many 
things. He makes a plan of observations and neqotiations. He 
arranges for various persons to observe the prograrr With 
their help he prepares brief narratives, portrayals, product 
displays, graphs, etc. He finds out what is of value to his 
audiences. He gathers expressions of worth from various 
individuals whose points of view differ. Of course, he checks 
the quality o f his records. He gets program personnel to react 
to the accuracy of his portrayals. He gets authority figures to 
react to the importance of various findings. He gets audience 
members to react to the relevance of his findings. Hg does 
much of this informally, iterating, and keeping a record of 
action and reaction. He chooses media accessible to his 
audiences to increase the liklihood and fidelity of communication. 
He might prepare a final written report; he might not— depending 
on what he and his clients have agreed on. ( ) 
Responsive evaluation allows the evaluator to respond to both current and 
emarging issues, and to adjust easily to changing conditions. A major focus 
is on observing the program in action. 

Responsive evaluation is preferred in formative evaluation when monitoring 
is desired and no particular problems are projected. It is preferred for 
summative evaluation when what is desired is an understanding of the program's 
activities, strengths, and shortcomings. Preordinate evaluation (discussed 
previously) is preferred when the purpose is to see if goals have been reached 
or promises kept, and to test hypotheses and issues that were* predetermined. 
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Figure 4 



A TWO-DIMENSIONAL MATRIX THAT RELATES 
STUFFLEBEAH'S FOUR TYPES Of DECISIONS AND ASSOCIATED EVALUATION TYPES 3 



FOCUSES ON 
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End Results 



Means of Obtaining Results 



Plann i ng Decisi ons--Decisinns about the 
setting to be served, the "ogram ends or out- 
comes intended, and whether or not program 
mission, goals, and objectives should be 
changed. 



Cont ext Evalu ation — Developing a rationale for 
determination of program objectives, i dent i - 
firacion of potential methodological 
strategies, anj develop- ent of proposals for 
outsiue funding through identification and 
analysis of: (1) needs and opportunities; 
(2) problems and constraints related to 
those needs and opportunities; (3) discre- 
pancies between actual and intended inputs and 
outputs; (4) basei .ne information regarding 
actual, probable, and possible program 
operations and accor.pl ishments; (5) internal 
and external environment philosophies, 
values, attitudes, noals, priorities, 
politics, economics, demographics, traditions, 
practices, etc.; (6) technological advances 
in the field, and (7) strategies, ouerations, 
and results obtained in similar programs 
• elsewhere. 



j Recycling Deci sions--Deci sions to continue, 
• terminate, refine, revise, or refocus the 
I program; based on the attainments achieved. 



Structuri ng Decisi ons—Decisions about 
sources of support "and the means to be used 
to achieve the desired outcomes, including 
decisions about methods, content, organi- 
zation, personnel, schedule, facilities, 
and budget. 



In put Evalua tion —Determining if outside 
assistance is required to meet the program 
objectives, how the objectives should be 
selected operationally, the overall program 
strategy to employ, and the best use of 
available resources to meet program goals 
effectively and efficiently through identifi- 
cation and analysis of- (1) availability of 
human and material resources and capabilities 
for the program; (2) sources of possible 
additional resources, likelihood of obtaining 
such support, and the probable kinds and 
amounts of support; (3) relevance, effec- 
tiveness, feasibility, a»id economy of 
alternative solution strategies and 
procedural designs that are available to 
the program; and (4) numbers and charac- 
teristics of entities--such as students or 
vther client groups—that are intended to 
be served, acted on, and/or modified by 
the program. 



Pr oces s Evaluation-- Identi fying and pre- 
dicting (based on "program operation) defects 
j in procedural design or its implementation, 
1 servicing the operational decisions built 
into the program desiQn, and determining the 
extent to which program procedures are 
operating as intended, through: (1) con- 
tinuously monitoring and identifying staff and 
student interactions, ^mr.iunication channels, 
logistics, adequacy 0i program resour ,s, 
amount of consensus among program staff and 
participants about the purposes of the 
program, sources of problems, unanticipated 
bottlenecks and other problems, etc.; (2) 
providing the information needed to make the 
operational decisions specified by the program 
plan; and (3) recording program process 
events and activities as they ocrjr, and 
relating them to what vmi projected in the 
program design, for indications of why the 
program objectives are or are not being m?t. 



Impleme ntin g Deci sions — Opera t i ona 1 
decisions pertafrTing to utilization, control, 
modification, and refinement of the program 
procedures and design. 



Product Ey_aluation--Mca$uring and intepreting 
program process "and outcome attainments 
durir.g and at the end of each program cycle, 
through: (1) defininq program objecti ves 
1n concrete, observab e, and operational 
terms; (2) identifying or developing 
indicators and measures, and associated 
interpretational criteria; (3) collecting 
data for the indicators and measures speci- 
fied; (4) cor\ann<| the measurement and 
indicator dau to either absolute or 
comparative standards selected as the 
criteria to be met, (5) interpreting 
the outcome results obtained, through 
relating them to the context, input, and 
process information gathered in the other 
three types of evaluation; and (6) making 
judgments about program "Orth and/or how 
program outcones Can be improved through 
program mod i f i cat ion. 



Abstracted from Stufflebeam e^aj_ (1971), page 79-84 and 218-235. 
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Stufflebeam's Model 

Stufflebeam and his associates (1971) introduced a new definition and focus 
for evaluation: Evaluation is the proces c of delineating, obtaining and pro- 
viding useful information for judging decision alternatives/' In addition, they 
set forth fou^ separate types of evaluation, each of which primarily influences 
one of four major types of decisions as shown below: 



Evaluation Type 



Decision Type 



Context Evaluation 
Input Evaluation • 
Process Evaluation 
Product Evaluation 



-^Plann.ng Decisions j 
-^Structuring Decisions j 
-^Implementing Decisions j 
-^Recycling Decisions ; 



What each type of evaluation and each type of decision involves is described in 
Figure 4. In this model, continual feedback is being provided to the decision- 
maker, which may cause him/her to reconsider earlier decisions. 



Figure 4 goes about here 

An additional word should be said about th* context in which the program is 
operating and being administered— highl ighted as a part of Figure 3. That context 
also can markedly affect the evaluation i'self in ways that the evaluator must 
consider. For example, as discussed by Smith (1977), the politics and values that 
dominate the internal and external environments should be expected to have 
profound impacts on the evaluation process. 

According to Stufflebeam, the structure of tasks in the evaluation design 
is the same for all four types of evaluation. He presented those tasks as 
fol lows: 
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TASKS 

Focusing the Evaluation 

1. Identify the major level(s) of decision-making to be served, e.g., 
local, state, or national. 

2. For each level of decision-making, project the decision situations 
to be served and describe each one in terms of its locus, focus, 
critical ity, timing, and composition of alternatives. 

3. Define criteria for each decision situation by specifying variables 
for measurement and standards for use in the judgment of alternatives 

4. Define policies within which the evaluator must operate. 
Collection of Information 

1. Specify the source of the information to be collected. 

2. Specify the instruments and methods for collecting the needed 
information. 

3. Specify the sampling procedure to be employed. 

4. Specify the conditions and schedule for information collection. 
Organization of Information 

1. Provide a format for the information which is to be collected. 

2. Designate a means for performing the analysis. 
Analysis of Information 

1. Select the analytical procedures to be employed. 

2. Designate a means for performing the analysis. 
Reporting of Information 

1. Define the audiences for the evaluation reports. 

2. Specify means for providing information to the audiences. 

3. Specify the format for evaluation reports and/or reporting sessions. 

4. Schedule the reporting of information. 
Administration of the Evaluation 

1. Summarize the evaluation schedule. 

2. Define staff and resource requirements and plans for meeting these 
requirements. 
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3. Specify means for meeting policy requirements for conduct of the 
eval uation. 

4. Evaluate the potential of the evaluation design for providing 
information which is valid, reliable, credible, timely, and pervasive. 

5. Specify and schedule means for periodic updating of the evaluation 
design. 

6. Provide a budget for the total evaluation program. 
Stufflebeam and his associates (1971, pp. 27-30) discussed eleven criteria 

that evaluation should meet. These criteria, and what they imply are as 
follows: (1) internal validity— close correspondence between the evaluative 
information and the phenomena it represents, (2) external val idity— general izabil ity 
of the information, (3) reliability— consistency and repl icability of the infor- 
mat'on, (4) objectivity— pub! icness of the infomation, (5) relevance— purposes of 
the evaluation that are served, (6) importance— high priority information highlighted, 
(7) scope— comprehensiveness, (8) credibil ity- -amount of trust and integrity in the 
evaluation and evaluator perceived by pertinent others, (9) time! iness— information 
provided when needed, (10) pervasiveness— eval uative findings disseminated to all 
persons who need it, and (11) efficiency— eval uative time, cost, and personnel. 

A1 kin's Approa>. :. 

Alkins' (1969) Model is very similar to Stufflebeam' s in purpose and 
operation, but involves five types of e*'*l uation and five types of decisions. 
(He separates Stufflebeam's process evaluation into "program implementation" and 
"program improvement.") Each of the five types of evaluation is described below, 
and they occur in this order: 

1. Systems Assessment - status of the system on discrepancy needs 
and goals. 

2. Program Planning - matching of programs to needs. 

3. Program Implementation - extent to which the program has been 
introduced as intended to the group intended. 
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4, Program Improvement - information about program functioning, achieve- 
ment of objectives, and unintended outcomes which can guide program 
modification. 

5. Program Certification - worth and general izabil ity of the program to 
related situations. 

Hammond's Model 

Hammond (1973) has suggested an approach to evaluation of innovations, and 
makes use of a cube that represents an interaction of three dimensions which 
can be used to structure the evaluation. The "instructional dimension," as 
represented on one face of the cube, is divideo into the following categories 
and subcategories fo* an elementary school: 

A. Time (duration and sequence of blocks of time devoted to the clientele. 

B. Space (the vertical and horizontal organization of the clientele, 

1. Vertical and Horizontal Organization 

2. Content 

3. Methodology 

4. Facilities 

5. Cost 

A second face of the cube represents the "institutional dimension," which 
is split into the following categories for the elementary school: student, 
teacher, administrator, educational specialist, family, and community. Each 
of these groups that is involved in the process, is subdivided according to 
various descriptive variables, For example, Hammond subdivided "students" 
according to age, grade level, sex, familial variables, socioeconomic variables, 
physical health, mental health, achievement, ability, interest, and relationship 
to innovation. 
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"Behavioral dimension" is the final dimension proposed by Hammond. This 
dimension is broken into the Bloom taxonomy categories of cognitive, affective, 
and psychomotor. They were hoping to add a fourth category that they tentatively 
labeled "perceptual behavior." 

Hammond sa>: the following about his structure that is graphically represented 
by a cube: 

The structure developed provides a framework to produce factors 
that have a direct influence on a given innovation. The factors 
created by the interaction of one variable from each of the dimen- 
sions may be studied in any depth desired... In most cases, the study 
of a given factor will be determined by time, availability of tests 

and procedures, and the needs Once the forces affecting a given 

innovation have been identified and placed in a structure which permits 
an analysis of the interaction of these forces, the next step is that of 

placing the structure in a working model for evaluation Sound 

evaluation procedures require that the process begin with the current 
programs. Before attempts at innovation are made, adequate baseline 
data is required to make those decisions which determine the direction 
of the change process, (p. 167) 

Because of limited evaluation skills of personnel, Hammond proposes that the 
process at the beginning evaluate only one area of the insitution or program. With 
such a precaution, and effective training, he contends that local staff can 
conduct all aspects of the evaluation throughout the institution. 

Hammond's model involves six procedural steps that have been quoted and 
paraphrased as follows: 

1. Select an area(s) within the total program on which the evaluation 
should focus. 

2. For the area st cted, define descriptive variables for the 
instructional and institutional dimensions. 

3. State the objectives in behavioral terms, in terms of the behavior 
that will be accepted as evidence the client has achieved the 
objective, the conditions under which the behavior will be expected 
to occur, and the level of performance that is acceptable. Scriven 
(1971) made a statement that adds a relevant point here: "There are 
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non-behavioral objectives, but we have to translate them into behavioral 
objectives if we want to know whether we have achieved them. 

4. Assess the behavior described in the objectives. 

5. Analyze the results within factors and the relationships between 
factors, to arrive at conclusions based on actual behavior. 

6. Disseminate the conclusions and apply them to guiding the development 
of innovations in the program. 

Hammond's model utilizes the behavioral objective and feedback concepts 
developed in the 1 930 1 s by Ralph Tyler (1942), considered by many to be the "father 
educational evaluation." But it goes far beyond that to the application of a 
multi-dimensional structure showing factor interactions that can remind staff 
of important program factors that might be overlooked if the structure were not 
used. Of course Hammond's structure will need to be modified appropriately in 
order to be applied to the evaluation of student affairs programs, but the concepts, 
principles, and general procedures he outlines are pertinent in that setting. 

Provus 1 Discrepancy Evaluation 

"Discrepancy evaluation" is a term made popular by Provus (1971) in his 
model for evaluating ongoing educational programs. (The purpose he espoused 
for evaluation was to determine whether the program should be improved, maintained, 
or terminated.) His model involves three general procedures, plus five program 
development stages which have questions associated with each that need to be 
answered. The three general procedures are as follows: (1) agree upon and 
define program standards and objectives; (2) identify any discrepancies between 
observations made about particular aspects of the program and what should be the 
case according to the program standards and objectives; and (3) use the discrepancy 
information to identify program weaknesses and feed it back to the program 
developers to guide program modification and problem solving. As a basis for 



his model, Provus also focused on teamwork between the program developers and 
evaluators and the necessity for continuous communication between the evaluator 
and the program staff. 

The discrepancy evaluation concept can usefully be carried beyond the 
definition of discrepancy favored by Provus. For example, the current author 
is a strong proponent of profile analysis, where the patterns of similarities 
discrepancies among program factors within and across program dimensions are 
studied to identify not only weaknesses but strengths around which improvement 
can be built. Perhaps such discrepancy evaluation should be referred to as 
profile or pattern evaluation in order to distinguish it from the concept and 
model popularized by Provus. 

Other Evaluation Models 

Additional models have been proposed, with some specific concepts or ideas that 
deserve attention anu consideration. Worthen and Sanders (1973), for example, 
discuss the personal judgment or accreditation model that makes use of self -study, 
visitations, annual reports, and panels of expert judges. Although most institutions 
perhaps do not really try to use accreditation study data for program improvement, 
the accreditation bodies are proponents of such application of the data, and their 
rationale makes sense to this author. 

Wolf (1975) applied the concept of a jury trial to educational evaluation. 
In such a proceeding, r'idence is presented by advocates on either side to an 
impartial jury, which makes a judgment about the worth of the program. As in a 
jury trial, a "judge" is present to insure consistency and fairness in the 
proceedings. Wolf contends that his "judicial model" (or the "adversary model" 
as it is more commonly referred to) "demands that the evaluation focus on relevant 



192 

197 



and significant issues as determined by a broad variety of persons involved in 
or effected by the program (1975, page 186)." The adversary model has subsequently 
received mixed reviews (Arnstein, 1975; Popham and Carlson, 1977; Jackson, 1977; 
Thurston, 1978). 

Student affairs programs, as is true of other types of programs, consist 
of complex arrays of human players and a multitude of facte *s all interacting 
in an OR-going and dynamic process. Evaluatinq such interactions is an especially 
difficult task, and "transactional models" have especially focused on such 
dynamics (Rippy, 1973). Informal analyses, such as the case study, are the 
primary method of such models. Stake's responsive evaluation (discussed earlier) 
is, in effect, a transactional model. 

Another model to be mentioned is a paradigm involving multiple criterion 
measures for evaluating effectiveness that was synthesized by Metfessel and 
Michael (1967). They divided the evaluation process into eight phases or steps, 
and emphasized that judgmental decisions are involved in every phase: 

(1) Involve both directly and indirectly members of the total institutional 
community as participants in, and facilitators of, the program 
evaluation. This could also involve outside people such as parents and 
the community. 

(2) Formation of a cohesive paradigm of bread goals and specific objectives 
arranged in a hierarchical order from general to specific outcomes. 

(3) Translation of specific objectives into a communicable form applicable 
to facilitating learning in the institutional environment. 

(4) Gather or develop the instrumentation necessary for furnishing 
criterion measures from which inferences can be formulated concerning 
program effectiveness in terms of the objectives set forth. 

(5) Carry out periodic observations through the use of 'tests, scales, 
and other indices of behavioral change that are considered valid 
with respect to the objectives sampled. 



(6) Analyze the data furnished by the status and change measures through 
use of appropriate statistical methods. 

(7) Interpretation of the data relative to specific standards, objectives 
and broad goals. 

(8) Formulate recommendations for further implementation, modification, 
or revision of the program or its goals and objectives. 

As the various evaluation strategies have been discussed, this writer's 
personal views have crept in, for example, concerning the desirability of being on 
the lookout also for evidence of unintended outcome^ during the evaluation process, 
and che importance of periodically conducting an evaluation of one's program goals. 
One strategy strongly hJd by the author was not really emphasized by any of the 
theorists, however—regarding the desirability of a wholistic phase followed by a 
focused phase. The wholistic phase is diverse and "broad-band" in its focus and 
examines many different factors at one time. Focused and more in-depth or detailed 
"narrow-band" follow-up of obvious problem areas is then culled for as a part of 
the overall process. 

Categorizations of Model Types 

To review and summarize the various models that have been outlined, two 
classifications of evaluation model types that were found in the literature will 
now be discussed. Popham (1975) came to the conclusion that there were four 
general classes of educational evaluation models, as follows: 

1. Goal -attainment mode is— The Tyler, Hammonds, and Metfessel and Michael 
models fall into this category. 

2. Judgmental models emphasizing intrinsic criteria— The accreditation 
model and Stake's responsive models fall into this category. 

3. Judgmental models emphasizing extrinsic criteria— Tha Scriven 
formulation, the adversary model, and Stake's countenance (or preordinate) 
model fall into this category. 
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4. Decision-facilitation models— The Stufflebeam, Alkin, and discrepancy 
evaluation models fall in this category. 

In a more recent attempt to categorize evaluation models, House (1978) differ- 
entiated eight categories: systems analysis, behavioral objectives, decision-making, 
goal free, art criticism, accreditation, adversary, and transaction. Ha 
differentiated them on the basis of their proponents, their major audiences, 
what they assume consensus on, the methodology used, the outcome or purpose, and 
the typical questions asked. For example, the typical questions associated with 
each type are as follows (House, 1978, p. 12): 

• Systems Analyses Are the expected effects achieved? 

Can the effects be achieved more economically? 
What are the most efficient programs? 

• Behavioral Objectives Are the students achieving the objectives? 

Is the staff producing? 

• Decision Making Is the program effective? 

What parts are effective? 

• Goal Free-- What are all the effects? 

• Art Criticism Would a critic approve this program? 

» Accreditation How would professionals rate this program? 

• Adversary— What are the arguments for and against the 

program? 

• Transaction What does the program look like to different 

people? 

Based on his study of the various models, House reported that "the major 
elements in understanding the models are their ethics, their epistemologies, 
and their political ramifications. 11 All of the models were found to derive from 
the philosophy of liberalism, which "grew out of an attempt to rationalize and 
justify a market society," and which focuses on choice, individualism, and 
empiracism. Similarly all of the major models were found to be subjective in 
ethics. Then the models break out as shown in Figure 5. 

Figure 5 goes about here 
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Conclusion: Staff Skills and Competencies Needed 
for Student Affairs Program Assessment and Evaluatio n 

The preceding discussion of strategies and procedures for conducting 

assessment and evaluation has implied a number of skills and competencies that 

student affairs assessors and evaluators should have in order to be effective, 

with the requirements varying according to the model (s) endorsed. Some skills 

are important for all approaches, however, such as the ability to ask the 

important questions, to think logically, and to communicate effectivaly. 

Two literary works were found that focused exclusively on evaluation 

skills. Scriven (1971) focused on the skills needed by a professional program 

evaluator, and made the following statement: 

Evaluation requires almost all of the skills known to man in order 

to be done well; although it is also true that we will do it in our 

amateur kind of way, sometimes quite successfully, whenever we buy 

a new washing machine or decide on a new automobile. It's worth 

remembering that those tasks which we all do as individuals are done 

with extreme care and quite demonstrably better by semi -technical 

skilled personnel, ranging from Tom McCahil down to the staff of 

consumer reports. 

In his discussion, Scriven emphasized a number of soecific skills for 
becoming competent in evaluation. One concerned the ability to conceptualize, 
another concerned the ability to work together with others as a team. Scriven 
also concluded that some evaluation problems, but not a majority, require 
considerable sophistication in statistics. Other skills and competencies 
mentioned included the ability to do moral and political analysis; an under- 
standing of game theory; ability to formulate goals in a way that makes it 
possible to determine later if they were achieved; ability to form behavioral 
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I 

objectives, plus to clarify and communicate about them; ability to link what is 
| observed in field tests to language; general knowledge about the program being 

m evaluated and the practices relevant to that field; the ability to formulate 

alternative hypotheses; the ability to identify concretely and specifically what 
I one is evaluating; and the ability to evaluate options and to make balanced, 

practical judgments about their desirability and feasibility. Scriven claimed 
I that all of these skills are important, but that the most crucial one is the 

m one listed last above. 

Owens and Evans (1977) focused on the program evaluation skills needed 
I by busy administrators, who are probably mors representative of the typical 

student affairs worker than is a professional evaluator. Interestingly, the 
I skills Owens and Evans emphasize are similar in nature to the ones emphasized 

■ by Scriven, and they cover a broader array of skills: 

ft The ability to identify the purposes and audiences for one'? 
_ evaluation; 

■ ft The ability to prepare a basic description of the program, and its 

activities, to be evaluated; 

I ft The ability to refine educational objectives in terms of who will 

do the action, what the activity is, the criteria for judging 

■ successful objective attainment, and the conditions under which 
| the activity will be conducted; 

- » The ability to write worthwhile (clear, emphasizes important skills 

I and processes, and provides a challenge at the same time it is 

™ achievable) objectives, and to determine which objectives are most 

critical to evaluate; 

■ ft The ability to describe the resources and processes to be used in 

achieving one '.s .objective; 

I • The ability to specify the alternative decisions likely to be 

made about a program; 

■ ft The ability to state evaluation questions clearly and concretely; 

• The ability to establish evaluation guidelines consistent with 
I funding availability, local concerns, administrative pol icy, and 

■ ethical principles; 
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t The ajility to identify available re:ources for conducting the 
eval uation; 

t The ability to speciry pertinent data sources; 

t The ability to determine appropriate ways to measure selected 
processes and outcomes; 

t The ability to select and apply instruments in terms of reliability, 
validity, and usefulness; 

• The ability to establish and apply criteria for the selection of an 
evaluation specialist; 

• Th2 ability to prepare a basic evaluation plan for collecting, analyzing, 
and reporting data and transforming it into information; 

• The ability to make judgments regarding various types and formats for 
evaluation reporting; and 

• The ability to applv various types of evaluation findings. 

All of the above skills and competencies are important for the assessment 
and evaluation o J . : student affairs programs. However, the amount of emphasis on 
each depends on the assessment and evaluation model one chooses to guide such 
efforts, or the combinations of strategies and procedures from various models 
that are integrated into one's personal framework. For example, Stake's 
responsive evaluation depends little on the formal measurement of outcomes and 
on statistics, and the component skills needed to conduct such activities; and 
emphasizes such skills as interpersonal relations, negotiation, natural obser- 
vation and communication, and the ability to po^cray and narrate. Conversely, 
his preordinate evaluation emphasizes such things as developing formal 
objectives; transforming those objectives into concrete, systematic data 
requirements; selecting formal instruments and using them plus formal observation 
procedures to gather the needed data; and using analyses of various kinds 
(including sophisticated statistical techniques where appropriate) to compare 
actuality to intentions on antecedent, transaction, and outcome variables. 
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Concerning statistical analyses, Stake would not expect the evaluator to be a 
statistician but to know enough about statistics to be able to intelligently 
choose and communicate with a statistics expert when needed for the project, 
and to appropriately interpret and apply any statistical data that are gathered. 

Which assessment and evaluation approaches one chooses will depend on many 
factors, including the philosophy and skills of the evaluator. It is quite 
appropriate to use components from several different models to form ore's own 
eclectic model, as long as it is well thought out and logically sensible in 
relation to the evaluator, the program, and the context within which the evaluator 
and the program must operate. Furthermore, the same evaluator may often need 
different approaches for different programs and contexts. 

A final point should be emphasized. The comprehensiveness of assessment 
and evaluation that is implied in this chapter is an ideal that will often 
involve more than the time, the financial resources, and the political restraints 
present, will allow. Furthermore, it is crucial for the reader to understand 
that the total evaluation for a program should not be attempted all at once, 
and that feasibility (with respect to fiscal resources, time, staff expertise, 
the political environment, etc.) must be a primary consideration in designing 
the evaluation plan— aloi.g with the needs of the program, the purposes the 
evaluation is to serve, what methods and activities will be effective, and so 
forth. An ongoing program evaluation plan should be cyclical, with differential 
phases totaling several years before the cycle is repeated. During the year 
that a particular segment of the program is being focused in-depth, simple 
monitoring techniques such as those discussed by Hecht (1977) should be used 
to keep ones "finger on the pulse and gross health" of the other areas of 
program functioning. If assessment and evaluation activities are well -planned 
and spaced appropriately, such activities can contribute greatly to program 
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improvement, support, and accountability. If not, staff members, evaluators, 
and outside observers are likely to become disillusioned, distrustful, and/or 
sarcastic concerning the evaluation effort— and possibly also with respect 
to the operational program. No evaluation would probably be better than 
ineffective and misleading evaluation. 
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